Project

General

Profile

Actions

Bug #3934

closed

TC_sgsap_expl_imsi_det_noneps crashes osmo-msc

Added by dexter about 5 years ago. Updated almost 5 years ago.

Status:
Resolved
Priority:
Immediate
Assignee:
Category:
-
Target version:
-
Start date:
04/16/2019
Due date:
% Done:

100%

Resolution:
Spec Reference:

Description

This is presumably similar to #3930.

Tue Apr 16 09:56:08 2019 DMNCC <0004> mncc_sock.c:320 MNCC socket at /home/owner/mncc_sock
Tue Apr 16 09:56:08 2019 DLGLOBAL <0012> telnet_interface.c:104 Available via telnet 127.0.0.1 4254
Tue Apr 16 09:56:08 2019 DSMPP <000c> smpp_smsc.c:1017 SMPP at 0.0.0.0 2775
Tue Apr 16 09:56:08 2019 DLCTRL <0019> control_if.c:911 CTRL at 127.0.0.1 4255
Tue Apr 16 09:56:08 2019 DLGSUP <001c> gsup_client.c:73 GSUP connecting to 127.0.0.1:4222
Tue Apr 16 09:56:08 2019 DLMGCP <0022> mgcp_client.c:716 MGCP client: using endpoint domain '@mgw'
Tue Apr 16 09:56:08 2019 DLMGCP <0022> mgcp_client.c:791 MGCP GW connection: r=127.0.0.1:2427<->l=127.0.0.1:2727
Tue Apr 16 09:56:08 2019 DLSCCP <001f> sccp_user.c:397 OsmoMSC-A: Using SS7 instance 0, pc:0.23.1
Tue Apr 16 09:56:08 2019 DLSCCP <001f> sccp_user.c:415 OsmoMSC-A: Using AS instance as-clnt-OsmoMSC-A
Tue Apr 16 09:56:08 2019 DLSCCP <001f> sccp_user.c:420 OsmoMSC-A: Creating default route
Tue Apr 16 09:56:08 2019 DLSCCP <001f> sccp_user.c:476 OsmoMSC-A: Using ASP instance asp-clnt-OsmoMSC-A
Tue Apr 16 09:56:08 2019 DLSS7 <001e> osmo_ss7.c:471 0: Creating SCCP instance
Tue Apr 16 09:56:08 2019 DSGS <0011> sgs_server.c:185 SGs socket bound to r=NULL<->l=0.0.0.0:29118
Tue Apr 16 09:56:08 2019 DBSSAP <0010> a_iface.c:674 Initalizing SCCP connection to stp...
Tue Apr 16 09:56:09 2019 DLGSUP <001c> gsup_client.c:73 GSUP connecting to 127.0.0.1:4222
Tue Apr 16 09:56:10 2019 DLGSUP <001c> gsup_client.c:73 GSUP connecting to 127.0.0.1:4222
Tue Apr 16 09:56:10 2019 DLM3UA <0021> m3ua.c:634 asp-asp-clnt-OsmoMSC-A: Received NOTIFY Type State Change:AS Inactive ()
Tue Apr 16 09:56:10 2019 DLSS7 <001e> xua_default_lm_fsm.c:353 xua_default_lm(asp-clnt-OsmoMSC-A)[0x5607406a6fe0]{ACTIVE}: Ignoring primitive M-ASP_ACTIVE.confirm
Tue Apr 16 09:56:10 2019 DLM3UA <0021> m3ua.c:634 asp-asp-clnt-OsmoMSC-A: Received NOTIFY Type State Change:AS Active ()
Tue Apr 16 09:56:11 2019 DLGSUP <001c> gsup_client.c:73 GSUP connecting to 127.0.0.1:4222
Tue Apr 16 09:56:12 2019 DLGSUP <001c> gsup_client.c:73 GSUP connecting to 127.0.0.1:4222
Tue Apr 16 09:56:13 2019 DLGSUP <001c> gsup_client.c:73 GSUP connecting to 127.0.0.1:4222
Tue Apr 16 09:56:13 2019 DLCTRL <0019> control_if.c:554 accept()ed new CTRL connection from (r=127.0.0.1:38313<->l=127.0.0.1:4255)
Tue Apr 16 09:56:13 2019 DMNCC <0004> mncc_sock.c:275 MNCC Socket has connection with external call control application
Tue Apr 16 09:56:14 2019 DLGSUP <001c> gsup_client.c:73 GSUP connecting to 127.0.0.1:4222
Tue Apr 16 09:56:14 2019 DLINP <0014> input/ipa.c:128 127.0.0.1:4222 connection done
Tue Apr 16 09:56:14 2019 DLINP <0014> input/ipaccess.c:705 received ID get from 0/0/0
Tue Apr 16 09:56:14 2019 DBSSAP <0010> a_iface.c:140 The calling BSC (RI=SSN_PC,PC=0.24.1,SSN=BSSAP) is unknown to this MSC ...
Tue Apr 16 09:56:14 2019 DBSSAP <0010> a_iface.c:490 Adding new BSC connection for BSC RI=SSN_PC,PC=0.24.1,SSN=BSSAP...
Tue Apr 16 09:56:14 2019 DBSSAP <0010> a_iface_bssap.c:112 Rx BSSMAP RESET from BSC RI=SSN_PC,PC=0.24.1,SSN=BSSAP, sending RESET ACK
Tue Apr 16 09:56:14 2019 DSMPP <000c> smpp_smsc.c:753 [] smpp_pdu_rx(00 00 00 32 00 00 00 09 00 00 00 00 00 00 00 01 6d 73 63 5f 74 65 73 74 65 72 00 6f 73 6d 6f 63 6f 6d 31 00 4d 53 43 5f 54 65 73 74 73 00 34 00 00 00 )
Tue Apr 16 09:56:14 2019 DSMPP <000c> smpp_smsc.c:546 [msc_tester] Rx BIND Trx (Version 34)
Tue Apr 16 09:56:14 2019 DSGS <0011> sgs_server.c:123 r=127.0.0.1:9999<->l=127.0.0.1:29118: Accepted new SGs connection
Tue Apr 16 09:56:14 2019 DLCTRL <0019> control_if.c:554 accept()ed new CTRL connection from (r=127.0.0.1:45709<->l=127.0.0.1:4255)
Tue Apr 16 09:56:14 2019 DSGS <0011> fsm.c:423 SGs-VLR-RESET(901-70-0001-01)[0x5607406ac120]{unknown 0}: Allocated
Tue Apr 16 09:56:14 2019 DSGS <0011> fsm.c:423 SGs-UE(num:0)[0x5607406aca20]{SGs-NULL}: Allocated
Tue Apr 16 09:56:14 2019 DSGS <0011> vlr_sgs_fsm.c:359 SGs-UE(num:0)[0x5607406aca20]{SGs-NULL}: state_chg to SGs-NULL
Tue Apr 16 09:56:14 2019 DREF <000a> vlr_sgs.c:83 VLR subscr unknown + SGs: now used by 1 (SGs)
Tue Apr 16 09:56:14 2019 DVLR <000e> vlr.c:446 set IMSI on subscriber; IMSI=262420000011815 id=262420000011815
Tue Apr 16 09:56:14 2019 DVLR <000e> vlr.c:397 New subscr, IMSI: 262420000011815
Tue Apr 16 09:56:14 2019 DVLR <000e> vlr.c:446 set IMSI on subscriber; IMSI=262420000011815 id=262420000011815
Tue Apr 16 09:56:14 2019 DSGS <0011> vlr_sgs.c:96 SGs-UE(num:0)[0x5607406aca20]{SGs-NULL}: Received Event RX_LU_FROM_MME
Tue Apr 16 09:56:14 2019 DSGS <0011> vlr_sgs_fsm.c:55 SGs-UE(num:0)[0x5607406aca20]{SGs-NULL}: state_chg to SGs-LA-UPDATE-PRESENT
Tue Apr 16 09:56:14 2019 DVLR <000e> gsm_04_08.c:1772 SUBSCR(IMSI-262420000011815:TMSInew-0x25E218F7) VLR: update for IMSI=262420000011815 (MSISDN=)
Tue Apr 16 09:56:14 2019 DVLR <000e> vlr.c:197 GSUP tx: 04010862420200001118f5280102
Tue Apr 16 09:56:14 2019 DVLR <000e> vlr.c:1092 GSUP rx 20: 10010862420200001118f5080706942103108151
Tue Apr 16 09:56:14 2019 DREF <000a> vlr.c:1113 VLR subscr IMSI-262420000011815:TMSInew-0x25E218F7 + vlr_gsupc_read_cb: now used by 2 (SGs,vlr_gsupc_read_cb)
Tue Apr 16 09:56:14 2019 DVLR <000e> vlr.c:800 IMSI:262420000011815 has MSISDN:491230011815
Tue Apr 16 09:56:14 2019 DVLR <000e> gsm_04_08.c:1772 SUBSCR(IMSI-262420000011815:MSISDN-491230011815:TMSInew-0x25E218F7) VLR: update for IMSI=262420000011815 (MSISDN=491230011815)
Tue Apr 16 09:56:14 2019 DVLR <000e> vlr.c:197 GSUP tx: 12010862420200001118f5
Tue Apr 16 09:56:14 2019 DREF <000a> vlr.c:1161 VLR subscr IMSI-262420000011815:MSISDN-491230011815:TMSInew-0x25E218F7 - vlr_gsupc_read_cb: now used by 1 (SGs)
Tue Apr 16 09:56:14 2019 DVLR <000e> vlr.c:1092 GSUP rx 11: 06010862420200001118f5
Tue Apr 16 09:56:14 2019 DREF <000a> vlr.c:1113 VLR subscr IMSI-262420000011815:MSISDN-491230011815:TMSInew-0x25E218F7 + vlr_gsupc_read_cb: now used by 2 (SGs,vlr_gsupc_read_cb)
Tue Apr 16 09:56:14 2019 DSGS <0011> vlr_sgs.c:116 SGs-UE(num:0)[0x5607406aca20]{SGs-LA-UPDATE-PRESENT}: Received Event TX_LU_ACCEPT
Tue Apr 16 09:56:14 2019 DSGS <0011> vlr_sgs_fsm.c:141 SGs-UE(imsi:262420000011815)[0x5607406aca20]{SGs-LA-UPDATE-PRESENT}: state_chg to SGs-ASSOCIATED
Tue Apr 16 09:56:14 2019 DREF <000a> vlr.c:1161 VLR subscr IMSI-262420000011815:MSISDN-491230011815:TMSInew-0x25E218F7 - vlr_gsupc_read_cb: now used by 1 (SGs)
Tue Apr 16 09:56:14 2019 DREF <000a> vlr_sgs.c:223 VLR subscr IMSI-262420000011815:MSISDN-491230011815:TMSInew-0x25E218F7 + vlr_sgs_tmsi_reall_compl: now used by 2 (SGs,vlr_sgs_tmsi_reall_compl)
Tue Apr 16 09:56:14 2019 DSGS <0011> vlr_sgs.c:227 SGs-UE(imsi:262420000011815)[0x5607406aca20]{SGs-ASSOCIATED}: Received Event RX_TMSI_REALLOC
Tue Apr 16 09:56:14 2019 DSGS <0011> vlr_sgs_fsm.c:206 SGs-UE(imsi:262420000011815)[0x5607406aca20]{SGs-ASSOCIATED}: state_chg to SGs-ASSOCIATED
Tue Apr 16 09:56:14 2019 DREF <000a> vlr_sgs.c:228 VLR subscr IMSI-262420000011815:MSISDN-491230011815:TMSI-0x25E218F7 - vlr_sgs_tmsi_reall_compl: now used by 1 (SGs)
Tue Apr 16 09:56:17 2019 DREF <000a> vlr_sgs.c:140 VLR subscr IMSI-262420000011815:MSISDN-491230011815:TMSI-0x25E218F7 + vlr_sgs_imsi_detach: now used by 2 (SGs,vlr_sgs_imsi_detach)
Tue Apr 16 09:56:17 2019 DSGS <0011> vlr_sgs.c:166 SGs-UE(imsi:262420000011815)[0x5607406aca20]{SGs-ASSOCIATED}: Received Event RX_DETACH_IND_FROM_MME
Tue Apr 16 09:56:17 2019 DSGS <0011> vlr_sgs_fsm.c:72 SGs-UE(imsi:262420000011815)[0x5607406aca20]{SGs-ASSOCIATED}: state_chg to SGs-NULL
Tue Apr 16 09:56:17 2019 DREF <000a> vlr.c:1254 VLR subscr IMSI-262420000011815:MSISDN-491230011815:TMSI-0x25E218F7 - attached: now used by 1 (SGs,vlr_sgs_imsi_detach,-1*attached)
Assert failed _osmo_use_count_get_put(&(vsub)->use_count, "attached", -1, "vlr.c", 1254) == 0 vlr.c:1254
backtrace() returned 11 addresses
/usr/local/lib/libosmocore.so.12(osmo_panic+0xbb) [0x7f0dbf83a8db]
osmo-msc(+0x3dfc1) [0x56073f346fc1]
osmo-msc(+0x446ee) [0x56073f34d6ee]
osmo-msc(+0x3637b) [0x56073f33f37b]
osmo-msc(+0x36ccb) [0x56073f33fccb]
/usr/local/lib/libosmonetif.so.6(+0xa7e3) [0x7f0dbf4037e3]
/usr/local/lib/libosmocore.so.12(osmo_select_main+0x1f1) [0x7f0dbf82fbc1]
osmo-msc(+0xd44f) [0x56073f31644f]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf1) [0x7f0dbe3c52b1]
osmo-msc(+0xd5ea) [0x56073f3165ea]
signal 6 received
backtrace() returned 15 addresses
osmo-msc(+0xd81d) [0x56073f31681d]
/lib/x86_64-linux-gnu/libc.so.6(+0x33030) [0x7f0dbe3d8030]
/lib/x86_64-linux-gnu/libc.so.6(gsignal+0xcf) [0x7f0dbe3d7fcf]
/lib/x86_64-linux-gnu/libc.so.6(abort+0x16a) [0x7f0dbe3d93fa]
/usr/local/lib/libosmocore.so.12(osmo_set_panic_handler+0) [0x7f0dbf83a8e0]
osmo-msc(+0x3dfc1) [0x56073f346fc1]
osmo-msc(+0x446ee) [0x56073f34d6ee]
osmo-msc(+0x3637b) [0x56073f33f37b]
osmo-msc(+0x36ccb) [0x56073f33fccb]
/usr/local/lib/libosmonetif.so.6(+0xa7e3) [0x7f0dbf4037e3]
/usr/local/lib/libosmocore.so.12(osmo_select_main+0x1f1) [0x7f0dbf82fbc1]
osmo-msc(+0xd44f) [0x56073f31644f]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf1) [0x7f0dbe3c52b1]
osmo-msc(+0xd5ea) [0x56073f3165ea]
talloc report on 'vty' (total 174968 bytes in 9344 blocks)
    struct vty                     contains    863 bytes in   4 blocks (ref 0) 0x5607406ab440
    struct vty                     contains   1004 bytes in  16 blocks (ref 0) 0x5607406a7450
    Configure SCCP timer values, see ITU-T Q.714
Waiting for connection confirm message, 1 to 2 minutes (default: 60)
Send keep-alive: on an idle connection, delay before sending an Idle Timer message, 5 to 10 minutes (default: 420)
Receive keep-alive: on an idle connection, delay until considering a connection as stale, 11 to 21 minutes (default: 900)
Waiting for release complete message, 10 to 20 seconds (default: 10)
Waiting for release complete message; or to repeat sending released message after the initial expiry, 10 to 20 seconds (default: 10)
Waiting for release complete message; or to release connection resources, freeze the LRN and alert a maintenance function after the initial expiry, extending to 1 minute (default: 60)
Waiting to resume normal procedure for temporary connection sections during the restart procedure, 23 to 25 minutes (default: 1380)
Waiting to release temporary connection section or alert maintenance function after reset request message is sent, 10 to 20 seconds (default: 10)
Waiting to receive all the segments of the remaining segments, single segmented message after receiving the first segment, 10 to 20 seconds (default: 10)
Timer value, in seconds
 contains   1194 bytes in   1 blocks (ref 0) 0x5607405c5830
    sccp-timer (conn_est|ias|iar|rel|repeat_rel|int|guard|reset|reassembly) <1-999999> contains     83 bytes in   1 blocks (ref 0) 0x5607405c56c0
    save_cwd                       contains     37 bytes in   1 blocks (ref 0) 0x560740587960
    vty_command                    contains 105253 bytes in 5615 blocks (ref 0) 0x560740574c20
    vty_vector                     contains  66534 bytes in 3705 blocks (ref 0) 0x560740574bb0
full talloc report on 'osmo_msc' (total  18143 bytes in  98 blocks)
    telnet_connection              contains    177 bytes in   3 blocks (ref 0) 0x56074069e0f0
        struct telnet_connection       contains     88 bytes in   1 blocks (ref 0) 0x5607406ab380
        struct telnet_connection       contains     88 bytes in   1 blocks (ref 0) 0x5607406aa590
    struct osmo_ss7_instance       contains   2478 bytes in  29 blocks (ref 0) 0x56074069e650
        struct osmo_sccp_instance      contains    266 bytes in   3 blocks (ref 0) 0x5607406a6570
            struct osmo_sccp_user          contains     90 bytes in   2 blocks (ref 0) 0x5607406a7110
                OsmoMSC-A                      contains     10 bytes in   1 blocks (ref 0) 0x56074069ebd0
        struct osmo_ss7_as             contains    624 bytes in   7 blocks (ref 0) 0x56074069ee70
            as-clnt-OsmoMSC-A              contains     18 bytes in   1 blocks (ref 0) 0x56074069f360
            struct osmo_fsm_inst           contains    364 bytes in   4 blocks (ref 0) 0x56074069f040
                struct xua_as_fsm_priv         contains    104 bytes in   1 blocks (ref 0) 0x56074069f290
                XUA_AS(as-clnt-OsmoMSC-A)[0x56074069f040] contains     42 bytes in   1 blocks (ref 0) 0x56074069f1f0
                as-clnt-OsmoMSC-A              contains     18 bytes in   1 blocks (ref 0) 0x56074069f170
            as-clnt-OsmoMSC-A              contains     18 bytes in   1 blocks (ref 0) 0x56074069efc0
        struct osmo_ss7_asp            contains   1147 bytes in  14 blocks (ref 0) 0x56074069eaa0
            (r=127.0.0.1:2905<->l=127.0.0.1:41915) contains     39 bytes in   1 blocks (ref 0) 0x56074069ed70
            struct osmo_fsm_inst           contains    367 bytes in   4 blocks (ref 0) 0x5607406a5de0
                struct xua_asp_fsm_priv        contains    104 bytes in   1 blocks (ref 0) 0x5607406a64a0
                XUA_ASP(asp-clnt-OsmoMSC-A)[0x5607406a5de0] contains     44 bytes in   1 blocks (ref 0) 0x5607406a5f10
                asp-clnt-OsmoMSC-A             contains     19 bytes in   1 blocks (ref 0) 0x56074069e1d0
            struct osmo_stream_cli         contains    242 bytes in   2 blocks (ref 0) 0x5607406a4a00
                127.0.0.1                      contains     10 bytes in   1 blocks (ref 0) 0x5607406a4b50
            struct osmo_fsm_inst           contains    278 bytes in   4 blocks (ref 0) 0x5607406a6fe0
                struct lm_fsm_priv             contains      8 bytes in   1 blocks (ref 0) 0x5607406a7b00
                xua_default_lm(asp-clnt-OsmoMSC-A)[0x5607406a6fe0] contains     51 bytes in   1 blocks (ref 0) 0x5607406a57b0
                asp-clnt-OsmoMSC-A             contains     19 bytes in   1 blocks (ref 0) 0x5607406a5860
            127.0.0.1                      contains     10 bytes in   1 blocks (ref 0) 0x56074069e2c0
            asp-clnt-OsmoMSC-A             contains     19 bytes in   1 blocks (ref 0) 0x56074069e540
        struct osmo_ss7_route_table    contains    145 bytes in   4 blocks (ref 0) 0x56074069e7e0
            struct osmo_ss7_route          contains     82 bytes in   2 blocks (ref 0) 0x5607406a59f0
                as-clnt-OsmoMSC-A              contains     18 bytes in   1 blocks (ref 0) 0x5607406a7a80
            system                         contains      7 bytes in   1 blocks (ref 0) 0x56074069e4d0
    struct osmo_stream_srv_link    contains    352 bytes in   4 blocks (ref 0) 0x56074069c870
        struct sgs_connection          contains    256 bytes in   2 blocks (ref 0) 0x5607406a6c90
            struct osmo_stream_srv         contains    104 bytes in   1 blocks (ref 0) 0x5607406a9d20
        0.0.0.0                        contains      8 bytes in   1 blocks (ref 0) 0x56074069c930
    struct sgs_state               contains    741 bytes in   5 blocks (ref 0) 0x56074069c690
        struct sgs_mme_ctx             contains    365 bytes in   4 blocks (ref 0) 0x5607406ac050
            struct osmo_fsm_inst           contains    261 bytes in   3 blocks (ref 0) 0x5607406ac120
                SGs-VLR-RESET(901-70-0001-01)[0x5607406ac120] contains     46 bytes in   1 blocks (ref 0) 0x5607406ac250
                901-70-0001-01                 contains     15 bytes in   1 blocks (ref 0) 0x5607406ab760
    struct smsc                    contains    600 bytes in   3 blocks (ref 0) 0x560740689f20
        struct osmo_esme               contains    336 bytes in   1 blocks (ref 0) 0x5607406a6ad0
        struct osmo_smpp_acl           contains    112 bytes in   1 blocks (ref 0) 0x56074069f460
    struct gsm_network             contains   7961 bytes in  31 blocks (ref 0) 0x5607405c7580
        struct bsc_context             contains    441 bytes in   5 blocks (ref 0) 0x5607406aa3f0
            struct osmo_fsm_inst           contains    241 bytes in   3 blocks (ref 0) 0x5607406a7290
                A-RESET(bsc-193)[0x5607406a7290] contains     33 bytes in   1 blocks (ref 0) 0x5607406a73c0
                bsc-193                        contains      8 bytes in   1 blocks (ref 0) 0x5607406a6d90
            struct reset_ctx               contains     16 bytes in   1 blocks (ref 0) 0x5607406aa510
        struct mgcp_client             contains    688 bytes in   1 blocks (ref 0) 0x5607406a4ff0
        struct gsm_sms_queue           contains    216 bytes in   1 blocks (ref 0) 0x5607406a4830
        struct ctrl_handle             contains    478 bytes in   5 blocks (ref 0) 0x56074069cfa0
            struct ctrl_connection         contains    199 bytes in   2 blocks (ref 0) 0x5607406ab1e0
                (r=127.0.0.1:45709<->l=127.0.0.1:4255) contains     39 bytes in   1 blocks (ref 0) 0x5607406ab2f0
            struct ctrl_connection         contains    199 bytes in   2 blocks (ref 0) 0x5607406a9b80
                (r=127.0.0.1:38313<->l=127.0.0.1:4255) contains     39 bytes in   1 blocks (ref 0) 0x5607406a9c90
        struct mncc_sock_state         contains    104 bytes in   1 blocks (ref 0) 0x56074069e880
        127.0.0.1                      contains     10 bytes in   1 blocks (ref 0) 0x5607405c8440
        /home/owner/mncc_sock          contains     22 bytes in   1 blocks (ref 0) 0x56074069e450
        112                            contains      4 bytes in   1 blocks (ref 0) 0x56074069e160
        127.0.0.1                      contains     10 bytes in   1 blocks (ref 0) 0x56074069e3d0
        OsmoMSC                        contains      8 bytes in   1 blocks (ref 0) 0x5607405c8360
        OsmoMSC                        contains      8 bytes in   1 blocks (ref 0) 0x5607405c83d0
        struct vlr_instance            contains   2804 bytes in  10 blocks (ref 0) 0x5607405c84c0
            struct vlr_subscr              contains   1994 bytes in   4 blocks (ref 0) 0x5607406ac2f0
                struct osmo_fsm_inst           contains    266 bytes in   3 blocks (ref 0) 0x5607406aca20
                    SGs-UE(imsi:262420000011815)[0x5607406aca20] contains     45 bytes in   1 blocks (ref 0) 0x5607406ad210
                    imsi:262420000011815           contains     21 bytes in   1 blocks (ref 0) 0x5607406ad190
            struct osmo_gsup_client        contains    490 bytes in   4 blocks (ref 0) 0x5607406a4340
                struct osmo_fd                 contains     48 bytes in   1 blocks (ref 0) 0x5607406a45d0
                struct ipa_client_conn         contains    186 bytes in   2 blocks (ref 0) 0x5607406a44b0
                    127.0.0.1                      contains     10 bytes in   1 blocks (ref 0) 0x5607406a4670
            struct ipaccess_unit           contains     64 bytes in   1 blocks (ref 0) 0x5607406a4290
        rate_ctr.c:234                 contains   2352 bytes in   1 blocks (ref 0) 0x5607405c7920
    logging                        contains   4393 bytes in   9 blocks (ref 0) 0x560740574360
        Configure logging
Set the log level for a specified category
A-bis Radio Link Layer (RLL)
Layer3 Call Control (CC)
Layer3 Mobility Management (MM)
Layer3 Radio Resource (RR)
MNCC API for Call Control application
Paging Subsystem
Mobile Switching Center
Media Gateway Control Protocol
Hand-Over
Database Layer
Reference Counting
Control interface
SMPP interface for external SMS apps
Radio Access Network Application Part Protocol
Visitor Location Register
Iu-CS Protocol
BSSAP Protocol (A Interface)
SGs Interface (SGsAP)
Library-internal global log family
LAPD in libosmogsm
A-bis Intput Subsystem
A-bis B-Subchannel TRAU Frame Multiplex
A-bis Input Driver for Signalling
A-bis Input Driver for B-Channels (voice)
Layer3 Short Message Service (SMS)
Control Interface
GPRS GTP library
Statistics messages and logging
Generic Subscriber Update Protocol
Osmocom Authentication Protocol
libosmo-sigtran Signalling System 7
libosmo-sigtran SCCP Implementation
libosmo-sigtran SCCP User Adaptation
libosmo-sigtran MTP3 User Adaptation
libosmo-mgcp Media Gateway Control Protocol
libosmo-netif Jitter Buffer
Remote SIM protocol
Deprecated alias for 'no logging level force-all'
 contains   1173 bytes in   1 blocks (ref 0) 0x5607405dcf70
        logging level (rll|cc|mm|rr|mncc|pag|msc|mgcp|ho|db|ref|ctrl|smpp|ranap|vlr|iucs|bssap|sgs|lglobal|llapd|linp|lmux|lmi|lmib|lsms|lctrl|lgtp|lstats|lgsup|loap|lss7|lsccp|lsua|lm3ua|lmgcp|ljibuf|lrspro) everything contains    212 bytes in   1 blocks (ref 0) 0x5607405dcd80
        Configure logging
Set the log level for a specified category
A-bis Radio Link Layer (RLL)
Layer3 Call Control (CC)
Layer3 Mobility Management (MM)
Layer3 Radio Resource (RR)
MNCC API for Call Control application
Paging Subsystem
Mobile Switching Center
Media Gateway Control Protocol
Hand-Over
Database Layer
Reference Counting
Control interface
SMPP interface for external SMS apps
Radio Access Network Application Part Protocol
Visitor Location Register
Iu-CS Protocol
BSSAP Protocol (A Interface)
SGs Interface (SGsAP)
Library-internal global log family
LAPD in libosmogsm
A-bis Intput Subsystem
A-bis B-Subchannel TRAU Frame Multiplex
A-bis Input Driver for Signalling
A-bis Input Driver for B-Channels (voice)
Layer3 Short Message Service (SMS)
Control Interface
GPRS GTP library
Statistics messages and logging
Generic Subscriber Update Protocol
Osmocom Authentication Protocol
libosmo-sigtran Signalling System 7
libosmo-sigtran SCCP Implementation
libosmo-sigtran SCCP User Adaptation
libosmo-sigtran MTP3 User Adaptation
libosmo-mgcp Media Gateway Control Protocol
libosmo-netif Jitter Buffer
Remote SIM protocol
Log debug messages and higher levels
Log informational messages and higher levels
Log noticeable messages and higher levels
Log error messages and higher levels
Log only fatal messages
 contains   1308 bytes in   1 blocks (ref 0) 0x5607405dc7f0
        logging level (rll|cc|mm|rr|mncc|pag|msc|mgcp|ho|db|ref|ctrl|smpp|ranap|vlr|iucs|bssap|sgs|lglobal|llapd|linp|lmux|lmi|lmib|lsms|lctrl|lgtp|lstats|lgsup|loap|lss7|lsccp|lsua|lm3ua|lmgcp|ljibuf|lrspro) (debug|info|notice|error|fatal) contains    233 bytes in   1 blocks (ref 0) 0x5607405dc600
        struct log_target              contains    242 bytes in   2 blocks (ref 0) 0x560740574970
            struct log_category            contains     74 bytes in   1 blocks (ref 0) 0x560740574a80
        struct log_info                contains   1224 bytes in   2 blocks (ref 0) 0x5607405743d0
            struct log_info_cat            contains   1184 bytes in   1 blocks (ref 0) 0x560740574460
    transaction                    contains      0 bytes in   1 blocks (ref 0) 0x5607405742f0
    gsm_call                       contains      0 bytes in   1 blocks (ref 0) 0x560740574280
    sms                            contains      0 bytes in   1 blocks (ref 0) 0x560740574210
    osmo_signal                    contains    280 bytes in   8 blocks (ref 0) 0x5607405741a0
        struct signal_handler          contains     40 bytes in   1 blocks (ref 0) 0x56074069ece0
        struct signal_handler          contains     40 bytes in   1 blocks (ref 0) 0x56074069ec50
        struct signal_handler          contains     40 bytes in   1 blocks (ref 0) 0x5607406aa840
        struct signal_handler          contains     40 bytes in   1 blocks (ref 0) 0x5607406a9690
        struct signal_handler          contains     40 bytes in   1 blocks (ref 0) 0x56074069e5c0
        struct signal_handler          contains     40 bytes in   1 blocks (ref 0) 0x56074069e340
        struct signal_handler          contains     40 bytes in   1 blocks (ref 0) 0x560740652e60
    msgb                           contains   1160 bytes in   2 blocks (ref 0) 0x560740574130
        SGsAP                          contains   1160 bytes in   1 blocks (ref 0) 0x5607406abb60
./start_msc.sh: line 6: 25552 Aborted                 osmo-msc -c ./osmo-msc.cfg
$ 

This time it was possible to reproduce the issue with gdb:

Tue Apr 16 09:58:30 2019 DSGS <0011> vlr_sgs_fsm.c:206 SGs-UE(imsi:262420000011815)[0x55555591b320]{SGs-ASSOCIATED}: state_chg to SGs-ASSOCIATED
Tue Apr 16 09:58:30 2019 DREF <000a> vlr_sgs.c:228 VLR subscr IMSI-262420000011815:MSISDN-491230011815:TMSI-0x3A4902EF - vlr_sgs_tmsi_reall_compl: now used by 1 (SGs)
Tue Apr 16 09:58:33 2019 DREF <000a> vlr_sgs.c:140 VLR subscr IMSI-262420000011815:MSISDN-491230011815:TMSI-0x3A4902EF + vlr_sgs_imsi_detach: now used by 2 (SGs,vlr_sgs_imsi_detach)
Tue Apr 16 09:58:33 2019 DSGS <0011> vlr_sgs.c:166 SGs-UE(imsi:262420000011815)[0x55555591b320]{SGs-ASSOCIATED}: Received Event RX_DETACH_IND_FROM_MME
Tue Apr 16 09:58:33 2019 DSGS <0011> vlr_sgs_fsm.c:72 SGs-UE(imsi:262420000011815)[0x55555591b320]{SGs-ASSOCIATED}: state_chg to SGs-NULL
Tue Apr 16 09:58:33 2019 DREF <000a> vlr.c:1254 VLR subscr IMSI-262420000011815:MSISDN-491230011815:TMSI-0x3A4902EF - attached: now used by 1 (SGs,vlr_sgs_imsi_detach,-1*attached)
Assert failed _osmo_use_count_get_put(&(vsub)->use_count, "attached", -1, "vlr.c", 1254) == 0 vlr.c:1254
backtrace() returned 11 addresses
/usr/local/lib/libosmocore.so.12(osmo_panic+0xbb) [0x7ffff731e8db]
/usr/local/bin/osmo-msc(+0x3dfc1) [0x555555591fc1]
/usr/local/bin/osmo-msc(+0x446ee) [0x5555555986ee]
/usr/local/bin/osmo-msc(+0x3637b) [0x55555558a37b]
/usr/local/bin/osmo-msc(+0x36ccb) [0x55555558accb]
/usr/local/lib/libosmonetif.so.6(+0xa7e3) [0x7ffff6ee77e3]
/usr/local/lib/libosmocore.so.12(osmo_select_main+0x1f1) [0x7ffff7313bc1]
/usr/local/bin/osmo-msc(+0xd44f) [0x55555556144f]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf1) [0x7ffff5ea92b1]
/usr/local/bin/osmo-msc(+0xd5ea) [0x5555555615ea]

Program received signal SIGABRT, Aborted.
__GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:51
51    ../sysdeps/unix/sysv/linux/raise.c: No such file or directory.
(gdb) bt
#0  __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:51
#1  0x00007ffff5ebd3fa in __GI_abort () at abort.c:89
#2  0x00007ffff731e8e0 in osmo_panic_default (args=0x7fffffffcb68, fmt=0x55555559a28c "Assert failed %s %s:%d\n") at panic.c:49
#3  osmo_panic (fmt=fmt@entry=0x55555559a28c "Assert failed %s %s:%d\n") at panic.c:84
#4  0x0000555555591fc1 in vlr_subscr_expire (vsub=vsub@entry=0x55555591abf0) at vlr.c:1254
#5  0x00005555555986ee in vlr_sgs_imsi_detach (vlr=<optimized out>, imsi=imsi@entry=0x7fffffffcca0 "262420000011815", type=SGSAP_ID_NONEPS_T_COMBINED_UE_EPS_NONEPS) at vlr_sgs.c:171
#6  0x000055555558a37b in sgs_rx_imsi_det_ind (tp=0x7fffffffce40, tp=0x7fffffffce40, imsi=0x7fffffffcca0 "262420000011815", msg=0x55555591a530, sgc=0x555555918650) at sgs_iface.c:634
#7  sgs_iface_rx (sgc=sgc@entry=0x555555918650, msg=msg@entry=0x55555591a530) at sgs_iface.c:985
#8  0x000055555558accb in sgs_conn_readable_cb (conn=0x555555913810) at sgs_server.c:87
#9  0x00007ffff6ee77e3 in osmo_stream_srv_read (conn=0x555555913810) at stream.c:894
#10 osmo_stream_srv_cb (ofd=<optimized out>, what=1) at stream.c:949
#11 0x00007ffff7313bc1 in osmo_fd_disp_fds (_eset=0x7fffffffe050, _wset=0x7fffffffdfd0, _rset=0x7fffffffdf50) at select.c:223
#12 osmo_select_main (polling=<optimized out>) at select.c:263
#13 0x000055555556144f in main (argc=3, argv=0x7fffffffe218) at msc_main.c:724
(gdb) 
Actions #1

Updated by dexter about 5 years ago

  • Assignee set to neels
Actions #2

Updated by neels about 5 years ago

  • Status changed from New to In Progress
  • Assignee changed from neels to dexter
  • Priority changed from Normal to Immediate

grepping for VSUB_USE_ATTACHED shows that the lu_complete flag corresponds to the VSUB_USE_ATTACHED use count.
Grepping for "lu_complete =" (and comparing current master to neels/ho) shows that a "get" is missing on current master in libvlr/vlr_sgs_fsm.c:131

Would you please merge such patch, since I am currently on vacation...

Actions #3

Updated by dexter about 5 years ago

Thanks! That shounds logical. I have tried that and now the SGs related tests succeed as well:

https://gerrit.osmocom.org/#/c/osmo-msc/+/13690 vlr_sgs_fsm: make sure vsub is marked used when LA is present

Actions #4

Updated by dexter about 5 years ago

  • % Done changed from 0 to 20

Unfortunately we see a problem here:

What happens when a subscriber attaches via SGs and attaches after that again from 2G, then the refcount would increment once more. Also for multiple attaches from the SGs interface there is no protection yet. For 2G there is a check on vsub->lu_complete so that the get happens only once.

Its probably also a good idea to have a TTCN3 for this.

Actions #5

Updated by dexter almost 5 years ago

  • Status changed from In Progress to Resolved

I have added a check to the patch to make sure that the refcount is not incremented for consecutive LUs. The patch is now merged and the SGs related TTCN3 tests are now passing again, which means the crash is fixed. So we can close this now.

Actions #6

Updated by dexter almost 5 years ago

  • % Done changed from 20 to 100
Actions

Also available in: Atom PDF

Add picture from clipboard (Maximum size: 48.8 MB)