Project

General

Profile

Actions
Copy link

Bug #4624

closed

osmo-bsc leaks memory

Added by fixeria over 3 years ago. Updated over 3 years ago.

Status:
Resolved
Priority:
High
Assignee:
pespin
Category:
-
Target version:
-
Start date:
06/20/2020
Due date:
% Done:

100%

Spec Reference:

Description

While investigating #4619, I noticed that osmo-bsc (or libosmo-abis?) leaks memory.

Before running LCLS test cases:


OsmoBSC# show talloc-context application brief 
talloc report on 'osmo-bsc' (total 914581 bytes in 584 blocks)
  telnet_connection              contains     89 bytes in   2 blocks (ref 0) 0x561a66e7a910
  0.0.0.0                        contains      8 bytes in   1 blocks (ref 0) 0x561a66e9a420
  struct osmo_ss7_instance       contains   3452 bytes in  28 blocks (ref 0) 0x561a66e7b6a0
  struct cmd_element             contains    122 bytes in   2 blocks (ref 0) 0x561a66e3c3a0
  struct cmd_element             contains    123 bytes in   2 blocks (ref 0) 0x561a66e3b410
  struct cmd_element             contains    121 bytes in   2 blocks (ref 0) 0x561a66e38860
  ../../../../src/libosmocore/src/vty/utils.c:353 contains    168 bytes in   1 blocks (ref 0) 0x561a66c6fd10
  ../../../../src/libosmocore/src/vty/utils.c:353 contains     56 bytes in   1 blocks (ref 0) 0x561a66c6fc70
  ../../../../src/libosmocore/src/vty/utils.c:353 contains    495 bytes in   1 blocks (ref 0) 0x561a66c6fa10
  ../../../../src/libosmocore/src/vty/utils.c:353 contains    130 bytes in   1 blocks (ref 0) 0x561a66c5a120
  abis                           contains 193781 bytes in  24 blocks (ref 0) 0x561a66c54630  // <--- check
  struct gsm_network             contains 709584 bytes in 488 blocks (ref 0) 0x561a66c53080
  logging                        contains   5971 bytes in  11 blocks (ref 0) 0x561a66c52880
  counter                        contains      0 bytes in   1 blocks (ref 0) 0x561a66c52810
  subch_txq_entry                contains      0 bytes in   1 blocks (ref 0) 0x561a66c527a0
  bs11_file_list_entry           contains      0 bytes in   1 blocks (ref 0) 0x561a66c52730
  paging_request                 contains      0 bytes in   1 blocks (ref 0) 0x561a66c526c0
  xua_msg                        contains      0 bytes in   1 blocks (ref 0) 0x561a66c52650
  osmo_signal                    contains    480 bytes in  13 blocks (ref 0) 0x561a66c525e0
  msgb                           contains      0 bytes in   1 blocks (ref 0) 0x561a66c52570

After running LCLS test cases:

OsmoBSC# show talloc-context application brief
talloc report on 'osmo-bsc' (total 1659989 bytes in 723 blocks)
  telnet_connection              contains     89 bytes in   2 blocks (ref 0) 0x560e7f96c910
  0.0.0.0                        contains      8 bytes in   1 blocks (ref 0) 0x560e7f98dcc0
  struct osmo_ss7_instance       contains   5326 bytes in  36 blocks (ref 0) 0x560e7f97af50
  struct cmd_element             contains    122 bytes in   2 blocks (ref 0) 0x560e7f92e3a0
  struct cmd_element             contains    123 bytes in   2 blocks (ref 0) 0x560e7f92d410
  struct cmd_element             contains    121 bytes in   2 blocks (ref 0) 0x560e7f92a860
  ../../../../src/libosmocore/src/vty/utils.c:353 contains    168 bytes in   1 blocks (ref 0) 0x560e7f761d10
  ../../../../src/libosmocore/src/vty/utils.c:353 contains     56 bytes in   1 blocks (ref 0) 0x560e7f761c70
  ../../../../src/libosmocore/src/vty/utils.c:353 contains    495 bytes in   1 blocks (ref 0) 0x560e7f761a10
  ../../../../src/libosmocore/src/vty/utils.c:353 contains    130 bytes in   1 blocks (ref 0) 0x560e7f74c120
  abis                           contains 869141 bytes in  66 blocks (ref 0) 0x560e7f746630  // <--- check
  struct gsm_network             contains 777226 bytes in 570 blocks (ref 0) 0x560e7f745080
  logging                        contains   6503 bytes in  18 blocks (ref 0) 0x560e7f744880
  counter                        contains      0 bytes in   1 blocks (ref 0) 0x560e7f744810
  subch_txq_entry                contains      0 bytes in   1 blocks (ref 0) 0x560e7f7447a0
  bs11_file_list_entry           contains      0 bytes in   1 blocks (ref 0) 0x560e7f744730
  paging_request                 contains      0 bytes in   1 blocks (ref 0) 0x560e7f7446c0
  xua_msg                        contains      0 bytes in   1 blocks (ref 0) 0x560e7f744650
  osmo_signal                    contains    480 bytes in  13 blocks (ref 0) 0x560e7f7445e0
  msgb                           contains      0 bytes in   1 blocks (ref 0) 0x560e7f744570

Here is a full report on the 'abis' chink:

OsmoBSC# show talloc-context application full tree 0x560e7f746630
full talloc report on 'osmo-bsc' (total 1659989 bytes in 723 blocks)
  abis                           contains 869141 bytes in  66 blocks (ref 0) 0x560e7f746630
    unixsocket                     contains      1 bytes in   1 blocks (ref 0) 0x560e7f746880
    ipa                            contains 820273 bytes in  56 blocks (ref 0) 0x560e7f746810
      struct e1inp_line              contains  48240 bytes in   3 blocks (ref 0) 0x560e7fa80dc0
        reference to: struct ipaccess_line 
        reference to: ../../../src/libosmocore/src/rate_ctr.c:234
      struct e1inp_line              contains  48240 bytes in   3 blocks (ref 0) 0x560e7fa73d20
        reference to: struct ipaccess_line 
        reference to: ../../../src/libosmocore/src/rate_ctr.c:234
      struct e1inp_line              contains  48240 bytes in   3 blocks (ref 0) 0x560e7fa68040
        reference to: struct ipaccess_line 
        reference to: ../../../src/libosmocore/src/rate_ctr.c:234
      struct e1inp_line              contains  48240 bytes in   3 blocks (ref 0) 0x560e7fa5c360
        reference to: struct ipaccess_line 
        reference to: ../../../src/libosmocore/src/rate_ctr.c:234
      struct e1inp_line              contains  48240 bytes in   3 blocks (ref 0) 0x560e7fa50680
        reference to: struct ipaccess_line 
        reference to: ../../../src/libosmocore/src/rate_ctr.c:234
      struct e1inp_line              contains  48240 bytes in   3 blocks (ref 0) 0x560e7fa449a0
        reference to: struct ipaccess_line 
        reference to: ../../../src/libosmocore/src/rate_ctr.c:234
      struct e1inp_line              contains  48240 bytes in   3 blocks (ref 0) 0x560e7fa38cc0
        reference to: struct ipaccess_line 
        reference to: ../../../src/libosmocore/src/rate_ctr.c:234
      struct e1inp_line              contains  48240 bytes in   3 blocks (ref 0) 0x560e7fa2bc20
        reference to: struct ipaccess_line 
        reference to: ../../../src/libosmocore/src/rate_ctr.c:234
      struct e1inp_line              contains  48240 bytes in   3 blocks (ref 0) 0x560e7fa1ff40
        reference to: struct ipaccess_line 
        reference to: ../../../src/libosmocore/src/rate_ctr.c:234
      struct e1inp_line              contains  48240 bytes in   3 blocks (ref 0) 0x560e7fa14260
        reference to: struct ipaccess_line 
        reference to: ../../../src/libosmocore/src/rate_ctr.c:234
      struct e1inp_line              contains  48240 bytes in   3 blocks (ref 0) 0x560e7fa08580
        reference to: struct ipaccess_line 
        reference to: ../../../src/libosmocore/src/rate_ctr.c:234
      struct e1inp_line              contains  48240 bytes in   3 blocks (ref 0) 0x560e7f9f6500
        reference to: struct ipaccess_line 
        reference to: ../../../src/libosmocore/src/rate_ctr.c:234
      struct e1inp_line              contains  48240 bytes in   3 blocks (ref 0) 0x560e7f9ea820
        reference to: struct ipaccess_line 
        reference to: ../../../src/libosmocore/src/rate_ctr.c:234
      struct e1inp_line              contains  48240 bytes in   3 blocks (ref 0) 0x560e7f9deb40
        reference to: struct ipaccess_line 
        reference to: ../../../src/libosmocore/src/rate_ctr.c:234
      struct e1inp_line              contains  48240 bytes in   3 blocks (ref 0) 0x560e7f9c9550
        reference to: struct ipaccess_line 
        reference to: ../../../src/libosmocore/src/rate_ctr.c:234
      struct e1inp_line              contains  48240 bytes in   3 blocks (ref 0) 0x560e7f9b26b0
        reference to: struct ipaccess_line 
        reference to: ../../../src/libosmocore/src/rate_ctr.c:234
      struct e1inp_line              contains  48240 bytes in   3 blocks (ref 0) 0x560e7f9a3380
        reference to: struct ipaccess_line 
        reference to: ../../../src/libosmocore/src/rate_ctr.c:234
      struct ipa_server_link         contains     96 bytes in   2 blocks (ref 0) 0x560e7f97bb30
        0.0.0.0                        contains      8 bytes in   1 blocks (ref 0) 0x560e7f99fb90
      struct ipa_server_link         contains     96 bytes in   2 blocks (ref 0) 0x560e7f97ba70
        0.0.0.0                        contains      8 bytes in   1 blocks (ref 0) 0x560e7f985340
    e1inp                          contains  48867 bytes in   8 blocks (ref 0) 0x560e7f7466a0
      struct e1inp_line              contains  48673 bytes in   3 blocks (ref 0) 0x560e7f96f050
        struct ipaccess_line           contains      1 bytes in   1 blocks (ref 17) 0x560e7f96d020
        ../../../src/libosmocore/src/rate_ctr.c:234 contains    432 bytes in   1 blocks (ref 17) 0x560e7f97ad30
      e1inp_sign_link                contains    193 bytes in   4 blocks (ref 0) 0x560e7f746710
        struct e1inp_sign_link         contains     64 bytes in   1 blocks (ref 0) 0x560e7f9c8520
        struct e1inp_sign_link         contains     64 bytes in   1 blocks (ref 0) 0x560e7f97b6a0
        struct e1inp_sign_link         contains     64 bytes in   1 blocks (ref 0) 0x560e7f97b7d0

Assigning to pespin (as discussed) since he was been working on reference counting recently.
Please see a capture file (containing GSMTAP logs, all debug) attached.

Files

osmo_bsc_memleak.log osmo_bsc_memleak.log 201 KB fixeria, 06/19/2020 07:22 PM
osmo_bsc_memleak.pcapng.gz osmo_bsc_memleak.pcapng.gz 4.1 MB fixeria, 06/19/2020 07:22 PM

Related issues

Related to OsmoBTS - Bug #3612: osmo-bts-trx: heap-use-after-free in e1inp_sign_link_destroyResolvedpespin10/02/2018

Actions
Related to OsmoBSC - Bug #4688: TC_chopped_ipa_ping causes use-after-free abort in ipaccess_drop() / msgb_free(e1i_ts->pending_msg); because of recent libosmo-abis commit "ipaccess: Drop e1inp_line reference in ipacess_drop()"Resolvedpespin07/31/2020

Actions
Actions
Copy link
 #1

Updated by pespin over 3 years ago

  • Related to Bug #3612: osmo-bts-trx: heap-use-after-free in e1inp_sign_link_destroy added
Actions
Copy link
 #2

Updated by pespin over 3 years ago

Most probably the issue appeared after fixing a crash in #3612:
https://gerrit.osmocom.org/c/libosmo-abis/+/18730 e1_input: refcount inc line during e1_sign_link_create, not during line update

Actions
Copy link
 #3

Updated by fixeria over 3 years ago

  • Priority changed from Normal to High

After running all test cases from ttcn3-bts-test, the 'abis' chunk alone occupies ~212MiB (!):

  abis                           contains 221856403 bytes in 13807 blocks (ref 0) 0x608000000580

Setting to 'High' because I think it's critical for setups in crowded places.

Actions
Copy link
 #4

Updated by pespin over 3 years ago

  • Status changed from New to Feedback
  • % Done changed from 0 to 90

Should be fixed by:
remote: https://gerrit.osmocom.org/c/libosmo-abis/+/19256 e1_input: Use osmo_use_count in e1inp_line
remote: https://gerrit.osmocom.org/c/libosmo-abis/+/19257 ipaccess: Drop e1inp_line reference in ipacess_drop()
remote: https://gerrit.osmocom.org/c/libosmo-abis/+/19258 ipacces: Fix e1inp_line reference put in ipaccess_close
remote: https://gerrit.osmocom.org/c/libosmo-abis/+/19259 ipaccess: Set bfd->data to NULL before releasing its reference
remote: https://gerrit.osmocom.org/c/libosmo-abis/+/19260 ipaccess_recvmsg: Clean up release steps upon error condition
remote: https://gerrit.osmocom.org/c/libosmo-abis/+/19261 ipaccess_recvmsg: Assert the new bfd from new line differs from the old one
remote: https://gerrit.osmocom.org/c/libosmo-abis/+/19262 ipaccess_recvmsg: Untangle code updating line

Actions
Copy link
 #5

Updated by neels over 3 years ago

  • Related to Bug #4688: TC_chopped_ipa_ping causes use-after-free abort in ipaccess_drop() / msgb_free(e1i_ts->pending_msg); because of recent libosmo-abis commit "ipaccess: Drop e1inp_line reference in ipacess_drop()" added
Actions
Copy link
 #6

Updated by pespin over 3 years ago

  • Status changed from Feedback to Resolved
  • % Done changed from 90 to 100

Merged, closing.

Actions
Copy link

Also available in: Atom PDF

Add picture from clipboard (Maximum size: 48.8 MB)