Project

General

Profile

Bug #4096

osmo-remsim-server leaking memory

Added by roh 12 days ago. Updated 4 days ago.

Status:
In Progress
Priority:
Urgent
Assignee:
Category:
-
Target version:
-
Start date:
07/10/2019
Due date:
% Done:

0%


Description

USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
root 13983 0.0 77.4 1051596 790600 ? Ssl Jun21 2:40 /usr/bin/osmo-remsim-server
~1 minute later:
root 13983 0.0 76.9 1051728 785484 ? Ssl Jun21 2:40 /usr/bin/osmo-remsim-server

this vm has 1gig of memory (which isn't much, but should be more than enough for osmo-remsim-server)

since this server process has just 4 clients and one simbank with 5 ports idling (not handling any requests yet) - this should not eat that amount of memory.


Related issues

Related to osmo-remsim - Bug #4097: osmo-remsim-bankd leaking memoryIn Progress07/10/2019

History

#1 Updated by roh 12 days ago

i just updated the packages on this vm and restarted the service.

freshly started service for comparison

USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
root     27625  2.5  0.5 120560  5828 ?        Ssl  18:11   0:00 /usr/bin/osmo-remsim-server
~ a minute later:
root     27625  0.0  0.5 120692  5828 ?        Ssl  18:11   0:00 /usr/bin/osmo-remsim-server
~ after connecting to the json interface and polling the client list 4 times:
root     27625  0.0  0.7 268156  7420 ?        Ssl  18:11   0:00 /usr/bin/osmo-remsim-server
root     27625  0.0  0.7 268288  7420 ?        Ssl  18:11   0:00 /usr/bin/osmo-remsim-server

#2 Updated by laforge 12 days ago

remsim-server has the usual 'SIGUSR1' handler of most osmocom projects, i.e. sending
a SIGURS1 to the process should render a talloc_report_full to stdout. From there
it's probably quite easy to see which objects are leaking.

#3 Updated by roh 11 days ago

i checked the talloc output but nothing stood out there:

Jul 11 14:41:45 remsimserver osmo-remsim-server[27625]: full talloc report on 'global' (total   6812 bytes in  61 blocks)
Jul 11 14:41:45 remsimserver osmo-remsim-server[27625]:     struct ipa_server_link         contains   2148 bytes in  32 blocks (ref 0) 0x559afafdc9a0
Jul 11 14:41:45 remsimserver osmo-remsim-server[27625]:         struct ipa_server_conn         contains    408 bytes in   6 blocks (ref 0) 0x559afafdd620
Jul 11 14:41:45 remsimserver osmo-remsim-server[27625]:             struct osmo_fsm_inst           contains    269 bytes in   4 blocks (ref 0) 0x559afafdd790
Jul 11 14:41:45 remsimserver osmo-remsim-server[27625]:                 IPA-KEEPALIVE(B1)[0x559afafdd790] contains     34 bytes in   1 blocks (ref 0) 0x559afafdd8c0
Jul 11 14:41:45 remsimserver osmo-remsim-server[27625]:                 B1                             contains      3 bytes in   1 blocks (ref 0) 0x559afb04b6a0
Jul 11 14:41:45 remsimserver osmo-remsim-server[27625]:                 struct ipa_fsm_priv            contains     32 bytes in   1 blocks (ref 0) 0x559afafdd950
Jul 11 14:41:45 remsimserver osmo-remsim-server[27625]:             10.9.99.30                     contains     11 bytes in   1 blocks (ref 0) 0x559afafdd710
Jul 11 14:41:45 remsimserver osmo-remsim-server[27625]:         struct ipa_server_conn         contains    411 bytes in   6 blocks (ref 0) 0x559afafe1b10
Jul 11 14:41:45 remsimserver osmo-remsim-server[27625]:             struct osmo_fsm_inst           contains    273 bytes in   4 blocks (ref 0) 0x559afafe1dc0
Jul 11 14:41:45 remsimserver osmo-remsim-server[27625]:                 IPA-KEEPALIVE(C0:2)[0x559afafe1dc0] contains     36 bytes in   1 blocks (ref 0) 0x559afafe1ef0
Jul 11 14:41:45 remsimserver osmo-remsim-server[27625]:                 C0:2                           contains      5 bytes in   1 blocks (ref 0) 0x559afafe3820
Jul 11 14:41:45 remsimserver osmo-remsim-server[27625]:                 struct ipa_fsm_priv            contains     32 bytes in   1 blocks (ref 0) 0x559afafe1f80
Jul 11 14:41:45 remsimserver osmo-remsim-server[27625]:             10.9.99.1                      contains     10 bytes in   1 blocks (ref 0) 0x559afafdea80
Jul 11 14:41:45 remsimserver osmo-remsim-server[27625]:         struct ipa_server_conn         contains    411 bytes in   6 blocks (ref 0) 0x559afafde910
Jul 11 14:41:45 remsimserver osmo-remsim-server[27625]:             struct osmo_fsm_inst           contains    273 bytes in   4 blocks (ref 0) 0x559afafe1580
Jul 11 14:41:45 remsimserver osmo-remsim-server[27625]:                 IPA-KEEPALIVE(C0:3)[0x559afafe1580] contains     36 bytes in   1 blocks (ref 0) 0x559afafe16b0
Jul 11 14:41:45 remsimserver osmo-remsim-server[27625]:                 C0:3                           contains      5 bytes in   1 blocks (ref 0) 0x559afafe3740
Jul 11 14:41:45 remsimserver osmo-remsim-server[27625]:                 struct ipa_fsm_priv            contains     32 bytes in   1 blocks (ref 0) 0x559afafe1740
Jul 11 14:41:45 remsimserver osmo-remsim-server[27625]:             10.9.99.1                      contains     10 bytes in   1 blocks (ref 0) 0x559afafdea00
Jul 11 14:41:45 remsimserver osmo-remsim-server[27625]:         struct ipa_server_conn         contains    411 bytes in   6 blocks (ref 0) 0x559afafdf4c0
Jul 11 14:41:45 remsimserver osmo-remsim-server[27625]:             struct osmo_fsm_inst           contains    273 bytes in   4 blocks (ref 0) 0x559afafdf770
Jul 11 14:41:45 remsimserver osmo-remsim-server[27625]:                 IPA-KEEPALIVE(C0:0)[0x559afafdf770] contains     36 bytes in   1 blocks (ref 0) 0x559afafdf8a0
Jul 11 14:41:45 remsimserver osmo-remsim-server[27625]:                 C0:0                           contains      5 bytes in   1 blocks (ref 0) 0x559afafe11d0
Jul 11 14:41:45 remsimserver osmo-remsim-server[27625]:                 struct ipa_fsm_priv            contains     32 bytes in   1 blocks (ref 0) 0x559afafdf930
Jul 11 14:41:45 remsimserver osmo-remsim-server[27625]:             10.9.99.1                      contains     10 bytes in   1 blocks (ref 0) 0x559afafdeed0
Jul 11 14:41:45 remsimserver osmo-remsim-server[27625]:         struct ipa_server_conn         contains    411 bytes in   6 blocks (ref 0) 0x559afafded10
Jul 11 14:41:45 remsimserver osmo-remsim-server[27625]:             struct osmo_fsm_inst           contains    273 bytes in   4 blocks (ref 0) 0x559afafdf060
Jul 11 14:41:45 remsimserver osmo-remsim-server[27625]:                 IPA-KEEPALIVE(C0:1)[0x559afafdf060] contains     36 bytes in   1 blocks (ref 0) 0x559afafdf190
Jul 11 14:41:45 remsimserver osmo-remsim-server[27625]:                 C0:1                           contains      5 bytes in   1 blocks (ref 0) 0x559afafe10f0
Jul 11 14:41:45 remsimserver osmo-remsim-server[27625]:                 struct ipa_fsm_priv            contains     32 bytes in   1 blocks (ref 0) 0x559afafdf220
Jul 11 14:41:45 remsimserver osmo-remsim-server[27625]:             10.9.99.1                      contains     10 bytes in   1 blocks (ref 0) 0x559afafdef50
Jul 11 14:41:45 remsimserver osmo-remsim-server[27625]:         0.0.0.0                        contains      8 bytes in   1 blocks (ref 0) 0x559afafdca60
Jul 11 14:41:45 remsimserver osmo-remsim-server[27625]:     struct rspro_server            contains   3775 bytes in  22 blocks (ref 0) 0x559afafdc7b0
Jul 11 14:41:45 remsimserver osmo-remsim-server[27625]:         struct osmo_fsm_inst           contains    235 bytes in   3 blocks (ref 0) 0x559afafe17d0
Jul 11 14:41:45 remsimserver osmo-remsim-server[27625]:             SERVER_CONN(B1)[0x559afafe17d0] contains     32 bytes in   1 blocks (ref 0) 0x559afafde700
Jul 11 14:41:45 remsimserver osmo-remsim-server[27625]:             B1                             contains      3 bytes in   1 blocks (ref 0) 0x559afb04b630
Jul 11 14:41:45 remsimserver osmo-remsim-server[27625]:         struct rspro_client_conn       contains    424 bytes in   1 blocks (ref 0) 0x559afafdd410
Jul 11 14:41:45 remsimserver osmo-remsim-server[27625]:         struct osmo_fsm_inst           contains    239 bytes in   3 blocks (ref 0) 0x559afafe1c00
Jul 11 14:41:45 remsimserver osmo-remsim-server[27625]:             SERVER_CONN(C0:2)[0x559afafe1c00] contains     34 bytes in   1 blocks (ref 0) 0x559afafe1d30
Jul 11 14:41:45 remsimserver osmo-remsim-server[27625]:             C0:2                           contains      5 bytes in   1 blocks (ref 0) 0x559afafe37b0
Jul 11 14:41:45 remsimserver osmo-remsim-server[27625]:         struct rspro_client_conn       contains    424 bytes in   1 blocks (ref 0) 0x559afafe1900
Jul 11 14:41:45 remsimserver osmo-remsim-server[27625]:         struct osmo_fsm_inst           contains    239 bytes in   3 blocks (ref 0) 0x559afafe1450
Jul 11 14:41:45 remsimserver osmo-remsim-server[27625]:             SERVER_CONN(C0:3)[0x559afafe1450] contains     34 bytes in   1 blocks (ref 0) 0x559afafdee00
Jul 11 14:41:45 remsimserver osmo-remsim-server[27625]:             C0:3                           contains      5 bytes in   1 blocks (ref 0) 0x559afafe36d0
Jul 11 14:41:45 remsimserver osmo-remsim-server[27625]:         struct rspro_client_conn       contains    424 bytes in   1 blocks (ref 0) 0x559afafe1240
Jul 11 14:41:45 remsimserver osmo-remsim-server[27625]:         struct osmo_fsm_inst           contains    239 bytes in   3 blocks (ref 0) 0x559afafdf5b0
Jul 11 14:41:45 remsimserver osmo-remsim-server[27625]:             SERVER_CONN(C0:0)[0x559afafdf5b0] contains     34 bytes in   1 blocks (ref 0) 0x559afafdf6e0
Jul 11 14:41:45 remsimserver osmo-remsim-server[27625]:             C0:0                           contains      5 bytes in   1 blocks (ref 0) 0x559afafe1160
Jul 11 14:41:45 remsimserver osmo-remsim-server[27625]:         struct rspro_client_conn       contains    424 bytes in   1 blocks (ref 0) 0x559afafdf2b0
Jul 11 14:41:45 remsimserver osmo-remsim-server[27625]:         struct osmo_fsm_inst           contains    239 bytes in   3 blocks (ref 0) 0x559afafde7e0
Jul 11 14:41:45 remsimserver osmo-remsim-server[27625]:             SERVER_CONN(C0:1)[0x559afafde7e0] contains     34 bytes in   1 blocks (ref 0) 0x559afafdefd0
Jul 11 14:41:45 remsimserver osmo-remsim-server[27625]:             C0:1                           contains      5 bytes in   1 blocks (ref 0) 0x559afafe1080
Jul 11 14:41:45 remsimserver osmo-remsim-server[27625]:         struct rspro_client_conn       contains    424 bytes in   1 blocks (ref 0) 0x559afafdeb00
Jul 11 14:41:45 remsimserver osmo-remsim-server[27625]:         struct slotmaps                contains     72 bytes in   1 blocks (ref 0) 0x559afafdcb20
Jul 11 14:41:45 remsimserver osmo-remsim-server[27625]:     logging                        contains    889 bytes in   5 blocks (ref 0) 0x559afafdc220
Jul 11 14:41:45 remsimserver osmo-remsim-server[27625]:         struct log_target              contains    208 bytes in   2 blocks (ref 0) 0x559afafdc610
Jul 11 14:41:45 remsimserver osmo-remsim-server[27625]:             struct log_category            contains     40 bytes in   1 blocks (ref 0) 0x559afafdc720
Jul 11 14:41:45 remsimserver osmo-remsim-server[27625]:         struct log_info                contains    680 bytes in   2 blocks (ref 0) 0x559afafdc290
Jul 11 14:41:45 remsimserver osmo-remsim-server[27625]:             struct log_info_cat            contains    640 bytes in   1 blocks (ref 0) 0x559afafdc320
Jul 11 14:41:45 remsimserver osmo-remsim-server[27625]:     asn1                           contains      0 bytes in   1 blocks (ref 0) 0x559afafdc1b0

so i went and ran valgrind --leak-check=yes on this and those were the most interesting details of multiple runs:

==28830== 304,220 bytes in 212 blocks are definitely lost in loss record 69 of 69
==28830==    at 0x4C2BBAF: malloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==28830==    by 0x4E3E661: talloc_named_const (in /usr/lib/x86_64-linux-gnu/libtalloc.so.2.1.8)
==28830==    by 0x52BFB61: msgb_alloc_c (in /usr/lib/x86_64-linux-gnu/libosmocore.so.12.1.0)
==28830==    by 0x50801C8: ipa_msg_alloc (in /usr/lib/x86_64-linux-gnu/libosmogsm.so.12.0.0)
==28830==    by 0x5080B3A: ipa_msg_recv_buffered (in /usr/lib/x86_64-linux-gnu/libosmogsm.so.12.0.0)
==28830==    by 0x54E927D: ??? (in /usr/lib/x86_64-linux-gnu/libosmoabis.so.6.0.2)
==28830==    by 0x52BF6F6: osmo_select_main (in /usr/lib/x86_64-linux-gnu/libosmocore.so.12.1.0)
==28830==    by 0x10B00E: main (remsim_server.c:63)

==28869== 14,350 bytes in 10 blocks are definitely lost in loss record 67 of 69
==28869==    at 0x4C2BBAF: malloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==28869==    by 0x4E3E661: talloc_named_const (in /usr/lib/x86_64-linux-gnu/libtalloc.so.2.1.8)
==28869==    by 0x52BFB61: msgb_alloc_c (in /usr/lib/x86_64-linux-gnu/libosmocore.so.12.1.0)
==28869==    by 0x50801C8: ipa_msg_alloc (in /usr/lib/x86_64-linux-gnu/libosmogsm.so.12.0.0)
==28869==    by 0x5080B3A: ipa_msg_recv_buffered (in /usr/lib/x86_64-linux-gnu/libosmogsm.so.12.0.0)
==28869==    by 0x54E927D: ??? (in /usr/lib/x86_64-linux-gnu/libosmoabis.so.6.0.2)
==28869==    by 0x52BF6F6: osmo_select_main (in /usr/lib/x86_64-linux-gnu/libosmocore.so.12.1.0)
==28869==    by 0x10B00E: main (remsim_server.c:63)

i stopped the systemd services for now and ran the server in screen and valgrind to see how what output i get tomorrow.

#4 Updated by roh 6 days ago

i ran it in valgrind for the weekend:

==28893==
==28893== 204,895,040 bytes in 142,784 blocks are definitely lost in loss record 70 of 70
==28893==    at 0x4C2BBAF: malloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==28893==    by 0x4E3E661: talloc_named_const (in /usr/lib/x86_64-linux-gnu/libtalloc.so.2.1.8)
==28893==    by 0x52BFB61: msgb_alloc_c (in /usr/lib/x86_64-linux-gnu/libosmocore.so.12.1.0)
==28893==    by 0x50801C8: ipa_msg_alloc (in /usr/lib/x86_64-linux-gnu/libosmogsm.so.12.0.0)
==28893==    by 0x5080B3A: ipa_msg_recv_buffered (in /usr/lib/x86_64-linux-gnu/libosmogsm.so.12.0.0)
==28893==    by 0x54E927D: ??? (in /usr/lib/x86_64-linux-gnu/libosmoabis.so.6.0.2)
==28893==    by 0x52BF6F6: osmo_select_main (in /usr/lib/x86_64-linux-gnu/libosmocore.so.12.1.0)
==28893==    by 0x10B00E: main (remsim_server.c:63)
==28893==
==28893== LEAK SUMMARY:
==28893==    definitely lost: 204,895,040 bytes in 142,784 blocks
==28893==    indirectly lost: 0 bytes in 0 blocks
==28893==      possibly lost: 26,269 bytes in 72 blocks
==28893==    still reachable: 131,121 bytes in 753 blocks
==28893==         suppressed: 0 bytes in 0 blocks
==28893== Reachable blocks (those to which a pointer was found) are not shown.
==28893== To see them, rerun with: --leak-check=full --show-leak-kinds=all
==28893==
==28893== For counts of detected and suppressed errors, rerun with: -v
==28893== ERROR SUMMARY: 29 errors from 29 contexts (suppressed: 0 from 0)
Killed

USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
root     28893  0.0 34.1 526252 348240 pts/2   Sl+  Jul11   2:49      \_ /usr/bin/valgrind.bin --leak-check=yes /usr/bin/osmo-remsim-server

#5 Updated by roh 6 days ago

for comparsion - this is a freshly started process in valgrind:

USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
root     31875 16.0  4.3 204472 44176 pts/2    Sl+  16:49   0:05      \_ /usr/bin/valgrind.bin --leak-check=yes --leak-check=full --show-leak-kinds=all /usr/bin/osmo-remsim-server

#6 Updated by Hoernchen 5 days ago

If Valgrind is right then this looks like something is not freeing the received msgbs after handling them, the ??? in libnosmo-abis is most likely the ipa server connection handling function, so i'd assume that the issue is in the read callback, rspro_server.c:sock_read_cb

#7 Updated by laforge 5 days ago

  • Assignee set to laforge
  • Priority changed from Normal to Urgent

#8 Updated by laforge 4 days ago

  • Status changed from New to In Progress

I guess https://gerrit.osmocom.org/c/osmo-remsim/+/14827 should fix this. roh please feel free to test.

#9 Updated by laforge 4 days ago

  • Related to Bug #4097: osmo-remsim-bankd leaking memory added

Also available in: Atom PDF

Add picture from clipboard (Maximum size: 48.8 MB)