Project

General

Profile

Actions

Bug #4874

closed

gbproxy: memory leak related to FR

Added by laforge over 3 years ago. Updated almost 3 years ago.

Status:
Resolved
Priority:
High
Assignee:
Target version:
-
Start date:
11/29/2020
Due date:
% Done:

100%

Spec Reference:

Description

After manually executing some TTCN3 test cases against osmo-gbproxy, I see the following memory leak in talloc reports:

msgb                           contains 291928 bytes in  92 blocks (ref 0) 0x608000000180
        Gb/NS/FR/GRE Rx                contains   3208 bytes in   1 blocks (ref 0) 0x61f0024928e0
        Gb/NS/FR/GRE Rx                contains   3208 bytes in   1 blocks (ref 0) 0x61f002491ae0
        Gb/NS/FR/GRE Rx                contains   3208 bytes in   1 blocks (ref 0) 0x61f002488ee0
        Gb/NS/FR/GRE Rx                contains   3208 bytes in   1 blocks (ref 0) 0x61f0024848e0
        Gb/NS/FR/GRE Rx                contains   3208 bytes in   1 blocks (ref 0) 0x61f002421ae0
        Gb/NS/FR/GRE Rx                contains   3208 bytes in   1 blocks (ref 0) 0x61f00241c6e0
        Gb/NS/FR/GRE Rx                contains   3208 bytes in   1 blocks (ref 0) 0x61f00241b8e0
        Gb/NS/FR/GRE Rx                contains   3208 bytes in   1 blocks (ref 0) 0x61f0024156e0
        Gb/NS/FR/GRE Rx                contains   3208 bytes in   1 blocks (ref 0) 0x61f0024148e0
        Gb/NS/FR/GRE Rx                contains   3208 bytes in   1 blocks (ref 0) 0x61f0023b8ae0
        Gb/NS/FR/GRE Rx                contains   3208 bytes in   1 blocks (ref 0) 0x61f0023b6ee0
        Gb/NS/FR/GRE Rx                contains   3208 bytes in   1 blocks (ref 0) 0x61f0023b36e0
        Gb/NS/FR/GRE Rx                contains   3208 bytes in   1 blocks (ref 0) 0x61f0023ae2e0
        Gb/NS/FR/GRE Rx                contains   3208 bytes in   1 blocks (ref 0) 0x61f00236f2e0
        Gb/NS/FR/GRE Rx                contains   3208 bytes in   1 blocks (ref 0) 0x61f00236c8e0
        Gb/NS/FR/GRE Rx                contains   3208 bytes in   1 blocks (ref 0) 0x61f00236bae0
        Gb/NS/FR/GRE Rx                contains   3208 bytes in   1 blocks (ref 0) 0x61f0023540e0
        Gb/NS/FR/GRE Rx                contains   3208 bytes in   1 blocks (ref 0) 0x61f0022d60e0
        Gb/NS/FR/GRE Rx                contains   3208 bytes in   1 blocks (ref 0) 0x61f0022d52e0
        Gb/NS/FR/GRE Rx                contains   3208 bytes in   1 blocks (ref 0) 0x61f0022d44e0
        Gb/NS/FR/GRE Rx                contains   3208 bytes in   1 blocks (ref 0) 0x61f0022d28e0
        Gb/NS/FR/GRE Rx                contains   3208 bytes in   1 blocks (ref 0) 0x61f00226d0e0
        Gb/NS/FR/GRE Rx                contains   3208 bytes in   1 blocks (ref 0) 0x61f00226c2e0
        Gb/NS/FR/GRE Rx                contains   3208 bytes in   1 blocks (ref 0) 0x61f00226b4e0
        Gb/NS/FR/GRE Rx                contains   3208 bytes in   1 blocks (ref 0) 0x61f00225d4e0
        Gb/NS/FR/GRE Rx                contains   3208 bytes in   1 blocks (ref 0) 0x61f0022556e0
        Gb/NS/FR/GRE Rx                contains   3208 bytes in   1 blocks (ref 0) 0x61f0022182e0
        Gb/NS/FR/GRE Rx                contains   3208 bytes in   1 blocks (ref 0) 0x61f0022166e0
        Gb/NS/FR/GRE Rx                contains   3208 bytes in   1 blocks (ref 0) 0x61f002214ae0
        Gb/NS/FR/GRE Rx                contains   3208 bytes in   1 blocks (ref 0) 0x61f00220b0e0
        Gb/NS/FR/GRE Rx                contains   3208 bytes in   1 blocks (ref 0) 0x61f0021beee0
        Gb/NS/FR/GRE Rx                contains   3208 bytes in   1 blocks (ref 0) 0x61f0021be0e0
        Gb/NS/FR/GRE Rx                contains   3208 bytes in   1 blocks (ref 0) 0x61f0021b0ee0
        Gb/NS/FR/GRE Rx                contains   3208 bytes in   1 blocks (ref 0) 0x61f0021b00e0
        Gb/NS/FR/GRE Rx                contains   3208 bytes in   1 blocks (ref 0) 0x61f00216f4e0
        Gb/NS/FR/GRE Rx                contains   3208 bytes in   1 blocks (ref 0) 0x61f00216e6e0
        Gb/NS/FR/GRE Rx                contains   3208 bytes in   1 blocks (ref 0) 0x61f00216d8e0
        Gb/NS/FR/GRE Rx                contains   3208 bytes in   1 blocks (ref 0) 0x61f00216bce0

What's even more odd is that I'm using NS/E1, and there is no GRE in this setup!

Actions #1

Updated by laforge over 3 years ago

  • Priority changed from High to Low

I haven't seen this anymore in recent days, and particularly with the major gbproxy rewrite that's going on, it probably doesn't mean we need to debug this anymore. Let's have a look after the new code has settled.

Actions #2

Updated by laforge about 3 years ago

  • Priority changed from Low to High

Unfortunately I'm again seeing memory leaks. osmo-gbproxy when idle after running a couple of TTCN3 tests (not even the full suite):

   msgb                           contains 239440 bytes in  75 blocks (ref 0) 0x608000000180
        GPRS/NS                        contains   3208 bytes in   1 blocks (ref 0) 0x61f00043a6e0
        GPRS/NS                        contains   3208 bytes in   1 blocks (ref 0) 0x61f0004398e0
        GPRS/NS                        contains   3208 bytes in   1 blocks (ref 0) 0x61f000436ee0
        GPRS/NS                        contains   3208 bytes in   1 blocks (ref 0) 0x61f0004360e0
        GPRS/NS                        contains   3208 bytes in   1 blocks (ref 0) 0x61f0004344e0
        GPRS/NS                        contains   3208 bytes in   1 blocks (ref 0) 0x61f0004328e0
        GPRS/NS                        contains   3208 bytes in   1 blocks (ref 0) 0x61f000430ce0
        GPRS/NS                        contains   3208 bytes in   1 blocks (ref 0) 0x61f00042e2e0
        GPRS/NS                        contains   3208 bytes in   1 blocks (ref 0) 0x61f00042c6e0
        GPRS/NS                        contains   3208 bytes in   1 blocks (ref 0) 0x61f00042aae0
        GPRS/NS                        contains   3208 bytes in   1 blocks (ref 0) 0x61f0003d4ee0
        GPRS/NS                        contains   3208 bytes in   1 blocks (ref 0) 0x61f0003d24e0
        GPRS/NS                        contains   3208 bytes in   1 blocks (ref 0) 0x61f0003d16e0
        GPRS/NS                        contains   3208 bytes in   1 blocks (ref 0) 0x61f0003d08e0
        GPRS/NS                        contains   3208 bytes in   1 blocks (ref 0) 0x61f0003cece0
        GPRS/NS                        contains   3208 bytes in   1 blocks (ref 0) 0x61f0003cd0e0
        GPRS/NS                        contains   3208 bytes in   1 blocks (ref 0) 0x61f0003cb4e0
        GPRS/NS                        contains   3208 bytes in   1 blocks (ref 0) 0x61f0003c98e0
        GPRS/NS                        contains   3208 bytes in   1 blocks (ref 0) 0x61f0003c7ce0
        GPRS/NS                        contains   3208 bytes in   1 blocks (ref 0) 0x61f0003c60e0
        GPRS/NS                        contains   3208 bytes in   1 blocks (ref 0) 0x61f0002d2ce0
        GPRS/NS                        contains   3208 bytes in   1 blocks (ref 0) 0x61f0002d1ee0
        GPRS/NS                        contains   3208 bytes in   1 blocks (ref 0) 0x61f0002cd8e0
        GPRS/NS                        contains   3208 bytes in   1 blocks (ref 0) 0x61f0002ccae0
        GPRS/NS                        contains   3208 bytes in   1 blocks (ref 0) 0x61f0002caee0
        GPRS/NS                        contains   3208 bytes in   1 blocks (ref 0) 0x61f0002c92e0
        GPRS/NS                        contains   3208 bytes in   1 blocks (ref 0) 0x61f0002c76e0
        GPRS/NS                        contains   3208 bytes in   1 blocks (ref 0) 0x61f0002c5ae0
        GPRS/NS                        contains   3208 bytes in   1 blocks (ref 0) 0x61f0002c3ee0
        GPRS/NS                        contains   3208 bytes in   1 blocks (ref 0) 0x61f0002c22e0
        GPRS/NS                        contains   3208 bytes in   1 blocks (ref 0) 0x61f0001ce0e0
        GPRS/NS                        contains   3208 bytes in   1 blocks (ref 0) 0x61f0001cd2e0
        GPRS/NS                        contains   3208 bytes in   1 blocks (ref 0) 0x61f0001ca8e0
        GPRS/NS                        contains   3208 bytes in   1 blocks (ref 0) 0x61f0001c9ae0
        GPRS/NS                        contains   3208 bytes in   1 blocks (ref 0) 0x61f0001c7ee0
        GPRS/NS                        contains   3208 bytes in   1 blocks (ref 0) 0x61f0001c62e0
        GPRS/NS                        contains   3208 bytes in   1 blocks (ref 0) 0x61f0001c46e0
        GPRS/NS                        contains   3208 bytes in   1 blocks (ref 0) 0x61f0001c2ae0
        GPRS/NS                        contains   3208 bytes in   1 blocks (ref 0) 0x61f0001c0ee0
        GPRS/NS                        contains   3208 bytes in   1 blocks (ref 0) 0x61f0001be4e0
        GPRS/NS                        contains   3208 bytes in   1 blocks (ref 0) 0x61f0001110e0
        GPRS/NS                        contains   3208 bytes in   1 blocks (ref 0) 0x61f0001102e0
        GPRS/NS                        contains   3208 bytes in   1 blocks (ref 0) 0x61f00010d8e0
        GPRS/NS                        contains   3208 bytes in   1 blocks (ref 0) 0x61f00010cae0
        GPRS/NS                        contains   3208 bytes in   1 blocks (ref 0) 0x61f00010aee0
        GPRS/NS                        contains   3208 bytes in   1 blocks (ref 0) 0x61f0001092e0
        GPRS/NS                        contains   3208 bytes in   1 blocks (ref 0) 0x61f0001076e0
        GPRS/NS                        contains   3208 bytes in   1 blocks (ref 0) 0x61f000105ae0
        GPRS/NS                        contains   3208 bytes in   1 blocks (ref 0) 0x61f000103ee0
        GPRS/NS                        contains   3208 bytes in   1 blocks (ref 0) 0x61f0001022e0
        GPRS/NS                        contains   3208 bytes in   1 blocks (ref 0) 0x61f0000896e0
        GPRS/NS                        contains   3208 bytes in   1 blocks (ref 0) 0x61f000086ce0
        GPRS/NS                        contains   3208 bytes in   1 blocks (ref 0) 0x61f000085ee0
        GPRS/NS                        contains   3208 bytes in   1 blocks (ref 0) 0x61f0000850e0
        GPRS/NS                        contains   3208 bytes in   1 blocks (ref 0) 0x61f0000834e0
        GPRS/NS                        contains   3208 bytes in   1 blocks (ref 0) 0x61f0000818e0
        GPRS/NS                        contains   3208 bytes in   1 blocks (ref 0) 0x61f00007fce0
        GPRS/NS                        contains   3208 bytes in   1 blocks (ref 0) 0x61f00007e0e0
        GPRS/NS                        contains   3208 bytes in   1 blocks (ref 0) 0x61f00007c4e0
        GPRS/NS                        contains   3208 bytes in   1 blocks (ref 0) 0x61f00007a8e0
        GPRS/NS                        contains   3208 bytes in   1 blocks (ref 0) 0x61f0000126e0
        GPRS/NS                        contains   3208 bytes in   1 blocks (ref 0) 0x61f0000118e0
        GPRS/NS                        contains   3208 bytes in   1 blocks (ref 0) 0x61f00000eee0
        GPRS/NS                        contains   3208 bytes in   1 blocks (ref 0) 0x61f00000e0e0
        GPRS/NS                        contains   3208 bytes in   1 blocks (ref 0) 0x61f00000c4e0
        GPRS/NS                        contains   3208 bytes in   1 blocks (ref 0) 0x61f00000a8e0
        GPRS/NS                        contains   3208 bytes in   1 blocks (ref 0) 0x61f000008ce0
        GPRS/NS                        contains   3208 bytes in   1 blocks (ref 0) 0x61f0000070e0
        GPRS/NS                        contains   3208 bytes in   1 blocks (ref 0) 0x61f0000054e0
        GPRS/NS                        contains   3208 bytes in   1 blocks (ref 0) 0x61f0000038e0
        GPRS/NS                        contains   3208 bytes in   1 blocks (ref 0) 0x61f000001ce0
        GPRS/NS                        contains   3208 bytes in   1 blocks (ref 0) 0x61f0000000e0
        BSSGP                          contains   4232 bytes in   1 blocks (ref 0) 0x621000002960
        BSSGP                          contains   4232 bytes in   1 blocks (ref 0) 0x621000001560
Actions #3

Updated by daniel about 3 years ago

  • Status changed from New to In Progress
  • Assignee changed from lynxis to daniel
Actions #4

Updated by daniel about 3 years ago

  • Status changed from In Progress to Feedback
  • Assignee changed from daniel to laforge
  • % Done changed from 0 to 60

Just found a leak in the SNS code, maybe that was it?

https://gerrit.osmocom.org/c/libosmocore/+/22255

Actions #5

Updated by laforge about 3 years ago

  • Status changed from Feedback to In Progress
  • Assignee changed from laforge to daniel

unfortunately I can still see the leak.

test setup:
  • GBProxy_Tests.ttcn against osmo-gbproxy over FR
  • manual execution of TC_flush_ll

after one test run:

   msgb                           contains  13856 bytes in   5 blocks (ref 0) 0x608000000180
        GPRS/NS                        contains   3208 bytes in   1 blocks (ref 0) 0x61f00000a8e0
        GPRS/NS                        contains   3208 bytes in   1 blocks (ref 0) 0x61f000009ae0
        GPRS/NS                        contains   3208 bytes in   1 blocks (ref 0) 0x61f000007ee0
        BSSGP                          contains   4232 bytes in   1 blocks (ref 0) 0x621000001560

after three test runs:

  msgb                           contains  42728 bytes in  14 blocks (ref 0) 0x608000000180
        GPRS/NS                        contains   3208 bytes in   1 blocks (ref 0) 0x61f002a33ae0
        GPRS/NS                        contains   3208 bytes in   1 blocks (ref 0) 0x61f002a32ce0
        GPRS/NS                        contains   3208 bytes in   1 blocks (ref 0) 0x61f002a310e0
        GPRS/NS                        contains   3208 bytes in   1 blocks (ref 0) 0x61f001c32ce0
        GPRS/NS                        contains   3208 bytes in   1 blocks (ref 0) 0x61f001c31ee0
        GPRS/NS                        contains   3208 bytes in   1 blocks (ref 0) 0x61f001c302e0
        GPRS/NS                        contains   3208 bytes in   1 blocks (ref 0) 0x61f000e2aee0
        GPRS/NS                        contains   3208 bytes in   1 blocks (ref 0) 0x61f000e2a0e0
        GPRS/NS                        contains   3208 bytes in   1 blocks (ref 0) 0x61f000e284e0
        GPRS/NS                        contains   3208 bytes in   1 blocks (ref 0) 0x61f00000c4e0
        GPRS/NS                        contains   3208 bytes in   1 blocks (ref 0) 0x61f00000b6e0
        GPRS/NS                        contains   3208 bytes in   1 blocks (ref 0) 0x61f000009ae0
        BSSGP                          contains   4232 bytes in   1 blocks (ref 0) 0x621000001560

so clearly there is some leakage of ~3 GPRS/NS msgb per test case execution. This was in a reduced configuration with only one NSE with one NSVC on the PCU side, and with only one SGSN with one NSE and two NSVC on the SGSN side. Given that this means three NS-VC in total, maybe there's a correlation.

Actions #6

Updated by laforge about 3 years ago

laforge wrote:

so clearly there is some leakage of ~3 GPRS/NS msgb per test case execution. This was in a reduced configuration with only one NSE with one NSVC on the PCU side, and with only one SGSN with one NSE and two NSVC on the SGSN side. Given that this means three NS-VC in total, maybe there's a correlation.

This looks plausible. When increasing again to three NSE with 4/2/2 NS-VC each, it looks like this after one test case execution:

    msgb                           contains  27712 bytes in   9 blocks (ref 0) 0x608000000180
        GPRS/NS                        contains   3208 bytes in   1 blocks (ref 0) 0x61f00002bce0
        GPRS/NS                        contains   3208 bytes in   1 blocks (ref 0) 0x61f0000284e0
        GPRS/NS                        contains   3208 bytes in   1 blocks (ref 0) 0x61f0000276e0
        GPRS/NS                        contains   3208 bytes in   1 blocks (ref 0) 0x61f0000268e0
        GPRS/NS                        contains   3208 bytes in   1 blocks (ref 0) 0x61f000024ce0
        GPRS/NS                        contains   3208 bytes in   1 blocks (ref 0) 0x61f0000230e0
Actions #7

Updated by lynxis about 3 years ago

  • Assignee changed from daniel to lynxis
  • % Done changed from 60 to 80
Actions #8

Updated by lynxis about 3 years ago

  • Status changed from In Progress to Feedback
  • % Done changed from 80 to 90
Actions #9

Updated by laforge almost 3 years ago

  • Status changed from Feedback to Resolved
  • % Done changed from 90 to 100

patch didn't contain a "Closes: OS..." line and henc this issue had to be manually closed.

Actions

Also available in: Atom PDF

Add picture from clipboard (Maximum size: 48.8 MB)