Actions
Bug #3206
closedosmo-bsc: heap-use-after-free at osmo_wqueue_bfd_cb
Start date:
04/24/2018
Due date:
% Done:
100%
Spec Reference:
Description
Catched by osmo-gsm-tester running test aoip_smpp:esme_ms_sms_transaction.py.
[0;m20180424135405115 [1;34mDLCTRL[0;m <0018> control_cmd.c:378 Command: GET bts.0.oml-connection-state
[0;m20180424135405117 [1;32mDLCTRL[0;m <0018> control_if.c:173 close()d CTRL connection (r=10.42.42.1:53908<->l=10.42.42.7:4249)
[0;m20180424135406115 [1;32mDLCTRL[0;m <0018> control_if.c:506 accept()ed new CTRL connection from (r=10.42.42.1:53910<->l=10.42.42.7:4249)
[0;m20180424135406116 [1;34mDLCTRL[0;m <0018> control_cmd.c:378 Command: GET bts.0.oml-connection-state
[0;m20180424135406117 [1;34mDLINP[0;m <0013> bts_ipaccess_nanobts.c:417 Identified BTS 1/0/0
[0;m[1;36m20180424135406118 [1;34mDNM[0;m[1;36m <0005> abis_nm.c:1628 Get Attr (bts=0)
[0;m[1;36m20180424135406118 [1;34mDNM[0;m[1;36m <0005> abis_nm.c:1628 Get Attr (bts=0)
[0;m20180424135406118 [1;34mDCTRL[0;m <000e> osmo_bsc_ctrl.c:158 BTS connection (re)established, sending TRAP.
[0;m20180424135406119 [1;32mDLCTRL[0;m <0018> control_if.c:173 close()d CTRL connection (r=10.42.42.1:53910<->l=10.42.42.7:4249)
[0;m=================================================================
==12301==ERROR: AddressSanitizer: heap-use-after-free on address 0x611000003e04 at pc 0x7f23091c3a2f bp 0x7ffc0cb73ff0 sp 0x7ffc0cb73fe8
READ of size 4 at 0x611000003e04 thread T0
#0 0x7f23091c3a2e in osmo_wqueue_bfd_cb /home/osmocom-build/jenkins/workspace/osmo-gsm-tester_build-osmo-bsc/libosmocore/src/write_queue.c:65
#1 0x7f23091ad5d8 in osmo_fd_disp_fds /home/osmocom-build/jenkins/workspace/osmo-gsm-tester_build-osmo-bsc/libosmocore/src/select.c:216
#2 0x7f23091ad5d8 in osmo_select_main /home/osmocom-build/jenkins/workspace/osmo-gsm-tester_build-osmo-bsc/libosmocore/src/select.c:256
#3 0x56538bdb7a26 in main /home/osmocom-build/jenkins/workspace/osmo-gsm-tester_build-osmo-bsc/osmo-bsc/src/osmo-bsc/osmo_bsc_main.c:532
#4 0x7f23077532e0 in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x202e0)
#5 0x56538bdb8999 in _start (/home/jenkins/workspace/osmo-gsm-tester_run-prod/trial-896/inst/osmo-bsc/bin/osmo-bsc+0x259999)
0x611000003e04 is located 132 bytes inside of 256-byte region [0x611000003d80,0x611000003e80)
freed by thread T0 here:
#0 0x7f230a1f8a10 in free (/usr/lib/x86_64-linux-gnu/libasan.so.3+0xc1a10)
#1 0x7f2309c2b86a in _talloc_free (/usr/lib/x86_64-linux-gnu/libtalloc.so.2+0x486a)
previously allocated by thread T0 here:
#0 0x7f230a1f8d28 in malloc (/usr/lib/x86_64-linux-gnu/libasan.so.3+0xc1d28)
#1 0x7f2309c2dacd in _talloc_zero (/usr/lib/x86_64-linux-gnu/libtalloc.so.2+0x6acd)
SUMMARY: AddressSanitizer: heap-use-after-free /home/osmocom-build/jenkins/workspace/osmo-gsm-tester_build-osmo-bsc/libosmocore/src/write_queue.c:65 in osmo_wqueue_bfd_cb
Shadow bytes around the buggy address:
0x0c227fff8770: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
0x0c227fff8780: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
0x0c227fff8790: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
0x0c227fff87a0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
0x0c227fff87b0: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
=>0x0c227fff87c0:[fd]fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
0x0c227fff87d0: fa fa fa fa fa fa fa fa fd fd fd fd fd fd fd fd
0x0c227fff87e0: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
0x0c227fff87f0: fd fd fd fd fd fd fd fd fa fa fa fa fa fa fa fa
0x0c227fff8800: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
0x0c227fff8810: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
Shadow byte legend (one shadow byte represents 8 application bytes):
Addressable: 00
Partially addressable: 01 02 03 04 05 06 07
Heap left redzone: fa
Heap right redzone: fb
Freed heap region: fd
Stack left redzone: f1
Stack mid redzone: f2
Stack right redzone: f3
Stack partial redzone: f4
Stack after return: f5
Stack use after scope: f8
Global redzone: f9
Global init order: f6
Poisoned by user: f7
Container overflow: fc
Array cookie: ac
Intra object redzone: bb
ASan internal: fe
Left alloca redzone: ca
Right alloca redzone: cb
==12301==ABORTING
Files
Related issues
Updated by pespin almost 6 years ago
Source code line triggering the report:
fd->when &= ~BSC_FD_WRITE;
It's the first time in the function were param "struct osmo_fd* fd" is used, which probably means we are freeing the "struct osmo_fd" somewhere without unregistering the fd from the wqueue.
Updated by pespin almost 6 years ago
The ctrl interface uses a wqueue. I think it's related to how the event happen in this case:
[0;m20180424135406115 [1;32mDLCTRL[0;m <0018> control_if.c:506 accept()ed new CTRL connection from (r=10.42.42.1:53910<->l=10.42.42.7:4249) [0;m20180424135406116 [1;34mDLCTRL[0;m <0018> control_cmd.c:378 Command: GET bts.0.oml-connection-state [0;m20180424135406117 [1;34mDLINP[0;m <0013> bts_ipaccess_nanobts.c:417 Identified BTS 1/0/0 [0;m[1;36m20180424135406118 [1;34mDNM[0;m[1;36m <0005> abis_nm.c:1628 Get Attr (bts=0) [0;m[1;36m20180424135406118 [1;34mDNM[0;m[1;36m <0005> abis_nm.c:1628 Get Attr (bts=0) [0;m20180424135406118 [1;34mDCTRL[0;m <000e> osmo_bsc_ctrl.c:158 BTS connection (re)established, sending TRAP. [0;m20180424135406119 [1;32mDLCTRL[0;m <0018> control_if.c:173 close()d CTRL connection (r=10.42.42.1:53910<->l=10.42.42.7:4249)
It seems we are sending a TRAP in between the time we already created the CTRL conn and we are closing it.
Updated by pespin almost 6 years ago
My first comment is actually not true. the struct osmo_fd is actually used in the parent function in the stack (osmo_fd_disp_fds) without libasan reporting any issue. This means the osmo_fd is freed by one of the previous callbacks in osmo_wqueue_bfd_cb before accessing again the osmo_fd struct:
if (what & BSC_FD_READ) {
rc = queue->read_cb(fd);
if (rc == -EBADF)
goto err_badfd;
}
if (what & BSC_FD_EXCEPT) {
rc = queue->except_cb(fd);
if (rc == -EBADF)
goto err_badfd;
}
The callbacks should imho return a specific err code to specify that the fd was released, and then the wqueue should stop using and forwarding it up the stack again.
Updated by pespin almost 6 years ago
- Status changed from New to Feedback
- % Done changed from 0 to 90
Should be fixed by https://gerrit.osmocom.org/#/c/8031/
Updated by pespin almost 6 years ago
- Status changed from Feedback to Resolved
- % Done changed from 90 to 100
Merged, closing.
Updated by pespin almost 6 years ago
- Related to Bug #3300: Avoid heap-use-after-free in osmo_wqueue_bfd_cb added
Actions
