Bug #4641
closed
osmo-ggsn: heap-use-after-free in sgsn_peer_drop_all_pdp_except
Added by pespin almost 4 years ago.
Updated almost 3 years ago.
Description
Got it a few seconds after killing (restarting) osmo-sgsn:
20200703132046337 DGGSN <0002> /git/osmo-ggsn/ggsn/ggsn.c:653 PDP(901700000015256:5): Packet received on APN(internet): forwarding to tun tun4
20200703132046348 DTUN <0001> /git/osmo-ggsn/ggsn/ggsn.c:632 TUN(tun4): Received packet for APN(internet)
20200703132047218 DGGSN <0002> /git/osmo-ggsn/ggsn/ggsn.c:653 PDP(901700000015256:5): Packet received on APN(internet): forwarding to tun tun4
20200703132047230 DTUN <0001> /git/osmo-ggsn/ggsn/ggsn.c:632 TUN(tun4): Received packet for APN(internet)
20200703132048335 DGGSN <0002> /git/osmo-ggsn/ggsn/ggsn.c:653 PDP(901700000015256:5): Packet received on APN(internet): forwarding to tun tun4
20200703132048346 DTUN <0001> /git/osmo-ggsn/ggsn/ggsn.c:632 TUN(tun4): Received packet for APN(internet)
20200703132049416 DGGSN <0002> /git/osmo-ggsn/ggsn/ggsn.c:653 PDP(901700000015256:5): Packet received on APN(internet): forwarding to tun tun4
20200703132049427 DTUN <0001> /git/osmo-ggsn/ggsn/ggsn.c:632 TUN(tun4): Received packet for APN(internet)
20200703132050417 DGGSN <0002> /git/osmo-ggsn/ggsn/ggsn.c:653 PDP(901700000015256:5): Packet received on APN(internet): forwarding to tun tun4
20200703132050429 DTUN <0001> /git/osmo-ggsn/ggsn/ggsn.c:632 TUN(tun4): Received packet for APN(internet)
20200703132051335 DGGSN <0002> /git/osmo-ggsn/ggsn/ggsn.c:653 PDP(901700000015256:5): Packet received on APN(internet): forwarding to tun tun4
20200703132051347 DTUN <0001> /git/osmo-ggsn/ggsn/ggsn.c:632 TUN(tun4): Received packet for APN(internet)
20200703132051636 DGGSN <0002> /git/osmo-ggsn/ggsn/sgsn.c:29 SGSN(127.0.0.1): Tx Echo Request
20200703132051636 DGGSN <0002> /git/osmo-ggsn/ggsn/sgsn.c:40 SGSN(127.0.0.1): Rx Echo Response
20200703132052318 DGGSN <0002> /git/osmo-ggsn/ggsn/ggsn.c:653 PDP(901700000015256:5): Packet received on APN(internet): forwarding to tun tun4
20200703132052330 DTUN <0001> /git/osmo-ggsn/ggsn/ggsn.c:632 TUN(tun4): Received packet for APN(internet)
20200703132053256 DGGSN <0002> /git/osmo-ggsn/ggsn/ggsn.c:653 PDP(901700000015256:5): Packet received on APN(internet): forwarding to tun tun4
20200703132053268 DTUN <0001> /git/osmo-ggsn/ggsn/ggsn.c:632 TUN(tun4): Received packet for APN(internet)
20200703132151636 DGGSN <0002> /git/osmo-ggsn/ggsn/sgsn.c:29 SGSN(127.0.0.1): Tx Echo Request
20200703132151637 DGGSN <0002> /git/osmo-ggsn/ggsn/sgsn.c:40 SGSN(127.0.0.1): Rx Echo Response
20200703132251637 DGGSN <0002> /git/osmo-ggsn/ggsn/sgsn.c:29 SGSN(127.0.0.1): Tx Echo Request
20200703132251637 DGGSN <0002> /git/osmo-ggsn/ggsn/sgsn.c:40 SGSN(127.0.0.1): Rx Echo Response
20200703132351637 DGGSN <0002> /git/osmo-ggsn/ggsn/sgsn.c:29 SGSN(127.0.0.1): Tx Echo Request
20200703132351637 DGGSN <0002> /git/osmo-ggsn/ggsn/sgsn.c:40 SGSN(127.0.0.1): Rx Echo Response
20200703132451637 DGGSN <0002> /git/osmo-ggsn/ggsn/sgsn.c:29 SGSN(127.0.0.1): Tx Echo Request
20200703132451637 DGGSN <0002> /git/osmo-ggsn/ggsn/sgsn.c:40 SGSN(127.0.0.1): Rx Echo Response
20200703132551637 DGGSN <0002> /git/osmo-ggsn/ggsn/sgsn.c:29 SGSN(127.0.0.1): Tx Echo Request
20200703132551637 DGGSN <0002> /git/osmo-ggsn/ggsn/sgsn.c:40 SGSN(127.0.0.1): Rx Echo Response
20200703132651637 DGGSN <0002> /git/osmo-ggsn/ggsn/sgsn.c:29 SGSN(127.0.0.1): Tx Echo Request
20200703132651638 DGGSN <0002> /git/osmo-ggsn/ggsn/sgsn.c:40 SGSN(127.0.0.1): Rx Echo Response
20200703132751637 DGGSN <0002> /git/osmo-ggsn/ggsn/sgsn.c:29 SGSN(127.0.0.1): Tx Echo Request
20200703132751638 DGGSN <0002> /git/osmo-ggsn/ggsn/sgsn.c:40 SGSN(127.0.0.1): Rx Echo Response
20200703132851638 DGGSN <0002> /git/osmo-ggsn/ggsn/sgsn.c:29 SGSN(127.0.0.1): Tx Echo Request
20200703132851638 DGGSN <0002> /git/osmo-ggsn/ggsn/sgsn.c:40 SGSN(127.0.0.1): Rx Echo Response
20200703132851638 DGGSN <0002> /git/osmo-ggsn/ggsn/sgsn.c:151 SGSN(127.0.0.1): SGSN recovery (174->175) pdp=(nil), releasing all PDP contexts
20200703132851638 DGGSN <0002> /git/osmo-ggsn/ggsn/ggsn.c:66 PDP(901700000015256:5): Sending DELETE PDP CTX due to shutdown
20200703132851638 DGGSN <0002> /git/osmo-ggsn/ggsn/ggsn.c:354 PDP(901700000015256:5): Deleting PDP context
20200703132851638 DGGSN <0002> /git/osmo-ggsn/ggsn/sgsn.c:21 SGSN(127.0.0.1): Deleting SGSN
20200703132851638 DLGTP <000d> /git/osmo-ggsn/gtp/pdp.c:296 Begin pdp_tiddel tid = 5652510000007109
20200703132851638 DLGTP <000d> /git/osmo-ggsn/gtp/pdp.c:303 End pdp_tiddel: PDP found
=================================================================
==12028==ERROR: AddressSanitizer: heap-use-after-free on address 0x611000005848 at pc 0x5555555da90f bp 0x7fffffffb7f0 sp 0x7fffffffb7e0
READ of size 8 at 0x611000005848 thread T0
#0 0x5555555da90e in sgsn_peer_drop_all_pdp_except /git/osmo-ggsn/ggsn/sgsn.c:123
#1 0x5555555db031 in sgsn_peer_handle_recovery /git/osmo-ggsn/ggsn/sgsn.c:157
#2 0x5555555d6a05 in cb_recovery3 /git/osmo-ggsn/ggsn/ggsn.c:782
#3 0x7ffff74fc66b in emit_cb_recovery /git/osmo-ggsn/gtp/gtp.c:223
#4 0x7ffff7508769 in gtp_echo_conf /git/osmo-ggsn/gtp/gtp.c:1134
#5 0x7ffff752a9e1 in gtp_decaps1c /git/osmo-ggsn/gtp/gtp.c:3154
#6 0x5555555d661e in ggsn_gtp_fd_cb /git/osmo-ggsn/ggsn/ggsn.c:725
#7 0x7ffff699ef76 in osmo_fd_disp_fds /git/libosmocore/src/select.c:227
#8 0x7ffff699f35b in _osmo_select_main /git/libosmocore/src/select.c:265
#9 0x7ffff699f43a in osmo_select_main /git/libosmocore/src/select.c:274
#10 0x5555555bb31c in main /git/osmo-ggsn/ggsn/ggsn_main.c:201
#11 0x7ffff5d3a001 in __libc_start_main (/usr/lib/libc.so.6+0x27001)
#12 0x5555555bab0d in _start (/build/new/out/bin/osmo-ggsn+0x66b0d)
0x611000005848 is located 136 bytes inside of 240-byte region [0x6110000057c0,0x6110000058b0)
freed by thread T0 here:
#0 0x7ffff766b0e9 in __interceptor_free /build/gcc/src/gcc/libsanitizer/asan/asan_malloc_linux.cpp:123
#1 0x7ffff689941b (/usr/lib/libtalloc.so.2+0x441b)
previously allocated by thread T0 here:
#0 0x7ffff766b459 in __interceptor_malloc /build/gcc/src/gcc/libsanitizer/asan/asan_malloc_linux.cpp:145
#1 0x7ffff689bb8c (/usr/lib/libtalloc.so.2+0x6b8c)
SUMMARY: AddressSanitizer: heap-use-after-free /git/osmo-ggsn/ggsn/sgsn.c:123 in sgsn_peer_drop_all_pdp_except
Shadow bytes around the buggy address:
0x0c227fff8ab0: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
0x0c227fff8ac0: fd fd fd fd fd fd fa fa fa fa fa fa fa fa fa fa
0x0c227fff8ad0: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
0x0c227fff8ae0: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fa fa
0x0c227fff8af0: fa fa fa fa fa fa fa fa fd fd fd fd fd fd fd fd
=>0x0c227fff8b00: fd fd fd fd fd fd fd fd fd[fd]fd fd fd fd fd fd
0x0c227fff8b10: fd fd fd fd fd fd fa fa fa fa fa fa fa fa fa fa
0x0c227fff8b20: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
0x0c227fff8b30: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
0x0c227fff8b40: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
0x0c227fff8b50: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
Shadow byte legend (one shadow byte represents 8 application bytes):
Addressable: 00
Partially addressable: 01 02 03 04 05 06 07
Heap left redzone: fa
Freed heap region: fd
Stack left redzone: f1
Stack mid redzone: f2
Stack right redzone: f3
Stack after return: f5
Stack use after scope: f8
Global redzone: f9
Global init order: f6
Poisoned by user: f7
Container overflow: fc
Array cookie: ac
Intra object redzone: bb
ASan internal: fe
Left alloca redzone: ca
Right alloca redzone: cb
Shadow gap: cc
==12028==ABORTING
[Inferior 1 (process 12028) exited with code 01]
(gdb)
Using osmo-ggsn.git 4e37fb356aafda0b12d8b33daa5057c43fe633f5
Failure line is:
llist_for_each_entry_safe(pdp, pdp2, &sgsn->pdp_list, entry) {
So it looks like some pdp context is left in the pdp_list after being freed (probably by libgtp?).
I got this today, potentially after restarting SGSN:
osmo-ggsn.git bd2b55679e897b8f2ef14bf24e4e17967098c03f
20210602190922315 DTUN <0001> /osmo-ggsn/ggsn/ggsn.c:643 TUN(tun4): APN(internet) Rx DL data packet for PDP(901700000015256:5): 176.16.222.5 <- 8.8.8.8
20210602190922352 DGGSN <0002> /osmo-ggsn/ggsn/ggsn.c:681 PDP(901700000015256:5): Packet received on APN(internet): forwarding to tun tun4
20210602190922363 DTUN <0001> /osmo-ggsn/ggsn/ggsn.c:643 TUN(tun4): APN(internet) Rx DL data packet for PDP(901700000015256:5): 176.16.222.5 <- 8.8.8.8
20210602190922365 DGGSN <0002> /osmo-ggsn/ggsn/ggsn.c:681 PDP(901700000015256:5): Packet received on APN(internet): forwarding to tun tun4
20210602190922376 DTUN <0001> /osmo-ggsn/ggsn/ggsn.c:643 TUN(tun4): APN(internet) Rx DL data packet for PDP(901700000015256:5): 176.16.222.5 <- 8.8.8.8
20210602190922404 DGGSN <0002> /osmo-ggsn/ggsn/ggsn.c:681 PDP(901700000015256:5): Packet received on APN(internet): forwarding to tun tun4
20210602190922415 DTUN <0001> /osmo-ggsn/ggsn/ggsn.c:643 TUN(tun4): APN(internet) Rx DL data packet for PDP(901700000015256:5): 176.16.222.5 <- 8.8.8.8
20210602190923042 DTUN <0001> /osmo-ggsn/ggsn/ggsn.c:643 TUN(tun4): APN(internet) Rx DL data packet for PDP(901700000015256:5): 176.16.222.5 <- 142.250.200.132
20210602190925090 DTUN <0001> /osmo-ggsn/ggsn/ggsn.c:643 TUN(tun4): APN(internet) Rx DL data packet for PDP(901700000015256:5): 176.16.222.5 <- 142.250.200.132
20210602190925490 DTUN <0001> /osmo-ggsn/ggsn/ggsn.c:643 TUN(tun4): APN(internet) Rx DL data packet for PDP(901700000015256:5): 176.16.222.5 <- 142.250.200.132
20210602190929122 DTUN <0001> /osmo-ggsn/ggsn/ggsn.c:643 TUN(tun4): APN(internet) Rx DL data packet for PDP(901700000015256:5): 176.16.222.5 <- 142.250.200.132
20210602190932390 DTUN <0001> /osmo-ggsn/ggsn/ggsn.c:643 TUN(tun4): APN(internet) Rx DL data packet for PDP(901700000015256:5): 176.16.222.5 <- 142.250.200.132
20210602190933810 DTUN <0001> /osmo-ggsn/ggsn/ggsn.c:643 TUN(tun4): APN(internet) Rx DL data packet for PDP(901700000015256:5): 176.16.222.5 <- 142.250.200.132
20210602190937378 DTUN <0001> /osmo-ggsn/ggsn/ggsn.c:643 TUN(tun4): APN(internet) Rx DL data packet for PDP(901700000015256:5): 176.16.222.5 <- 142.250.200.132
20210602190950194 DTUN <0001> /osmo-ggsn/ggsn/ggsn.c:643 TUN(tun4): APN(internet) Rx DL data packet for PDP(901700000015256:5): 176.16.222.5 <- 142.250.200.132
20210602190953763 DTUN <0001> /osmo-ggsn/ggsn/ggsn.c:643 TUN(tun4): APN(internet) Rx DL data packet for PDP(901700000015256:5): 176.16.222.5 <- 142.250.200.132
20210602190956947 DGGSN <0002> /osmo-ggsn/ggsn/sgsn.c:29 SGSN(127.0.0.1): Tx Echo Request
20210602190956947 DGGSN <0002> /osmo-ggsn/ggsn/sgsn.c:40 SGSN(127.0.0.1): Rx Echo Response
20210602191002599 DTUN <0001> /osmo-ggsn/ggsn/ggsn.c:643 TUN(tun4): APN(internet) Rx DL data packet for PDP(901700000015256:5): 176.16.222.5 <- 142.250.200.132
20210602191032807 DTUN <0001> /osmo-ggsn/ggsn/ggsn.c:643 TUN(tun4): APN(internet) Rx DL data packet for PDP(901700000015256:5): 176.16.222.5 <- 142.250.200.132
20210602191056947 DGGSN <0002> /osmo-ggsn/ggsn/sgsn.c:29 SGSN(127.0.0.1): Tx Echo Request
20210602191056947 DGGSN <0002> /osmo-ggsn/ggsn/sgsn.c:40 SGSN(127.0.0.1): Rx Echo Response
20210602191101364 DTUN <0001> /osmo-ggsn/ggsn/ggsn.c:643 TUN(tun4): APN(internet) Rx DL data packet for PDP(901700000015256:5): 176.16.222.5 <- 142.250.200.132
20210602191156947 DGGSN <0002> /osmo-ggsn/ggsn/sgsn.c:29 SGSN(127.0.0.1): Tx Echo Request
20210602191156947 DGGSN <0002> /osmo-ggsn/ggsn/sgsn.c:40 SGSN(127.0.0.1): Rx Echo Response
20210602191256947 DGGSN <0002> /osmo-ggsn/ggsn/sgsn.c:29 SGSN(127.0.0.1): Tx Echo Request
20210602191256947 DGGSN <0002> /osmo-ggsn/ggsn/sgsn.c:40 SGSN(127.0.0.1): Rx Echo Response
20210602191356947 DGGSN <0002> /osmo-ggsn/ggsn/sgsn.c:29 SGSN(127.0.0.1): Tx Echo Request
20210602191356947 DGGSN <0002> /osmo-ggsn/ggsn/sgsn.c:40 SGSN(127.0.0.1): Rx Echo Response
20210602191456947 DGGSN <0002> /osmo-ggsn/ggsn/sgsn.c:29 SGSN(127.0.0.1): Tx Echo Request
20210602191456947 DGGSN <0002> /osmo-ggsn/ggsn/sgsn.c:40 SGSN(127.0.0.1): Rx Echo Response
20210602191556947 DGGSN <0002> /osmo-ggsn/ggsn/sgsn.c:29 SGSN(127.0.0.1): Tx Echo Request
20210602191556947 DGGSN <0002> /osmo-ggsn/ggsn/sgsn.c:40 SGSN(127.0.0.1): Rx Echo Response
20210602191656947 DGGSN <0002> /osmo-ggsn/ggsn/sgsn.c:29 SGSN(127.0.0.1): Tx Echo Request
20210602191656948 DGGSN <0002> /osmo-ggsn/ggsn/sgsn.c:40 SGSN(127.0.0.1): Rx Echo Response
20210602191656948 DGGSN <0002> /osmo-ggsn/ggsn/sgsn.c:151 SGSN(127.0.0.1): SGSN recovery (62->63) pdp=(nil), releasing all PDP contexts
20210602191656948 DGGSN <0002> /osmo-ggsn/ggsn/ggsn.c:66 PDP(901700000015256:5): Sending DELETE PDP CTX due to shutdown
20210602191656948 DGGSN <0002> /osmo-ggsn/ggsn/ggsn.c:354 PDP(901700000015256:5): Deleting PDP context
20210602191656948 DGGSN <0002> /osmo-ggsn/ggsn/sgsn.c:21 SGSN(127.0.0.1): Deleting SGSN
20210602191656948 DLGTP <000d> /osmo-ggsn/gtp/pdp.c:296 Begin pdp_tiddel tid = 5652510000007109
20210602191656948 DLGTP <000d> /osmo-ggsn/gtp/pdp.c:303 End pdp_tiddel: PDP found
=================================================================
==221315==ERROR: AddressSanitizer: heap-use-after-free on address 0x611000006388 at pc 0x5555555df532 bp 0x7fffffffbaa0 sp 0x7fffffffba90
READ of size 8 at 0x611000006388 thread T0
#0 0x5555555df531 in sgsn_peer_drop_all_pdp_except /osmo-ggsn/ggsn/sgsn.c:123
#1 0x5555555dfc9e in sgsn_peer_handle_recovery /osmo-ggsn/ggsn/sgsn.c:157
#2 0x5555555db31d in cb_recovery3 /osmo-ggsn/ggsn/ggsn.c:810
#3 0x7ffff75007ab in emit_cb_recovery /osmo-ggsn/gtp/gtp.c:230
#4 0x7ffff750cf9a in gtp_echo_conf /osmo-ggsn/gtp/gtp.c:1141
#5 0x7ffff7530d83 in gtp_decaps1c /osmo-ggsn/gtp/gtp.c:3228
#6 0x5555555daf09 in ggsn_gtp_fd_cb /osmo-ggsn/ggsn/ggsn.c:753
#7 0x7ffff68299a1 in poll_disp_fds /libosmocore/src/select.c:350
#8 0x7ffff6829af6 in _osmo_select_main /libosmocore/src/select.c:378
#9 0x7ffff6829b15 in osmo_select_main /libosmocore/src/select.c:417
#10 0x5555555bd74e in main /osmo-ggsn/ggsn/ggsn_main.c:249
#11 0x7ffff5ba7b24 in __libc_start_main (/usr/lib/libc.so.6+0x27b24)
#12 0x5555555bcb3d in _start (/home/pespin/dev/sysmocom/build/new/out/bin/osmo-ggsn+0x68b3d)
20210602183719783 DGGSN <0002> /osmo-ggsn/ggsn/pco.c:205 PDP(901700000015256:5): PCO Protocol 0xc223
0x611000006388 is located 136 bytes inside of 240-byte region [0x611000006300,0x6110000063f0)
freed by thread T0 here:
#0 0x7ffff7676f19 in __interceptor_free /build/gcc/src/gcc/libsanitizer/asan/asan_malloc_linux.cpp:127
#1 0x7ffff671141b (/usr/lib/libtalloc.so.2+0x441b)
previously allocated by thread T0 here:
#0 0x7ffff7677279 in __interceptor_malloc /build/gcc/src/gcc/libsanitizer/asan/asan_malloc_linux.cpp:145
#1 0x7ffff6713b8c (/usr/lib/libtalloc.so.2+0x6b8c)
SUMMARY: AddressSanitizer: heap-use-after-free /osmo-ggsn/ggsn/sgsn.c:123 in sgsn_peer_drop_all_pdp_except
Shadow bytes around the buggy address:
0x0c227fff8c20: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fa fa
0x0c227fff8c30: fa fa fa fa fa fa fa fa fd fd fd fd fd fd fd fd
0x0c227fff8c40: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
0x0c227fff8c50: fd fd fd fd fd fd fa fa fa fa fa fa fa fa fa fa
0x0c227fff8c60: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
=>0x0c227fff8c70: fd[fd]fd fd fd fd fd fd fd fd fd fd fd fd fa fa
0x0c227fff8c80: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
0x0c227fff8c90: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
0x0c227fff8ca0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
0x0c227fff8cb0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
0x0c227fff8cc0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
Shadow byte legend (one shadow byte represents 8 application bytes):
Addressable: 00
Partially addressable: 01 02 03 04 05 06 07
Heap left redzone: fa
Freed heap region: fd
Stack left redzone: f1
Stack mid redzone: f2
Stack right redzone: f3
Stack after return: f5
Stack use after scope: f8
Global redzone: f9
Global init order: f6
Poisoned by user: f7
Container overflow: fc
Array cookie: ac
Intra object redzone: bb
ASan internal: fe
Left alloca redzone: ca
Right alloca redzone: cb
Shadow gap: cc
==221315==ABORTING
[Inferior 1 (process 221315) exited with code 01]
- Status changed from New to Feedback
- % Done changed from 0 to 90
- Status changed from Feedback to Resolved
- % Done changed from 90 to 100
Also available in: Atom
PDF