Bug #3206
closedosmo-bsc: heap-use-after-free at osmo_wqueue_bfd_cb
100%
Description
Catched by osmo-gsm-tester running test aoip_smpp:esme_ms_sms_transaction.py.
[0;m20180424135405115 [1;34mDLCTRL[0;m <0018> control_cmd.c:378 Command: GET bts.0.oml-connection-state [0;m20180424135405117 [1;32mDLCTRL[0;m <0018> control_if.c:173 close()d CTRL connection (r=10.42.42.1:53908<->l=10.42.42.7:4249) [0;m20180424135406115 [1;32mDLCTRL[0;m <0018> control_if.c:506 accept()ed new CTRL connection from (r=10.42.42.1:53910<->l=10.42.42.7:4249) [0;m20180424135406116 [1;34mDLCTRL[0;m <0018> control_cmd.c:378 Command: GET bts.0.oml-connection-state [0;m20180424135406117 [1;34mDLINP[0;m <0013> bts_ipaccess_nanobts.c:417 Identified BTS 1/0/0 [0;m[1;36m20180424135406118 [1;34mDNM[0;m[1;36m <0005> abis_nm.c:1628 Get Attr (bts=0) [0;m[1;36m20180424135406118 [1;34mDNM[0;m[1;36m <0005> abis_nm.c:1628 Get Attr (bts=0) [0;m20180424135406118 [1;34mDCTRL[0;m <000e> osmo_bsc_ctrl.c:158 BTS connection (re)established, sending TRAP. [0;m20180424135406119 [1;32mDLCTRL[0;m <0018> control_if.c:173 close()d CTRL connection (r=10.42.42.1:53910<->l=10.42.42.7:4249) [0;m================================================================= ==12301==ERROR: AddressSanitizer: heap-use-after-free on address 0x611000003e04 at pc 0x7f23091c3a2f bp 0x7ffc0cb73ff0 sp 0x7ffc0cb73fe8 READ of size 4 at 0x611000003e04 thread T0 #0 0x7f23091c3a2e in osmo_wqueue_bfd_cb /home/osmocom-build/jenkins/workspace/osmo-gsm-tester_build-osmo-bsc/libosmocore/src/write_queue.c:65 #1 0x7f23091ad5d8 in osmo_fd_disp_fds /home/osmocom-build/jenkins/workspace/osmo-gsm-tester_build-osmo-bsc/libosmocore/src/select.c:216 #2 0x7f23091ad5d8 in osmo_select_main /home/osmocom-build/jenkins/workspace/osmo-gsm-tester_build-osmo-bsc/libosmocore/src/select.c:256 #3 0x56538bdb7a26 in main /home/osmocom-build/jenkins/workspace/osmo-gsm-tester_build-osmo-bsc/osmo-bsc/src/osmo-bsc/osmo_bsc_main.c:532 #4 0x7f23077532e0 in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x202e0) #5 0x56538bdb8999 in _start (/home/jenkins/workspace/osmo-gsm-tester_run-prod/trial-896/inst/osmo-bsc/bin/osmo-bsc+0x259999) 0x611000003e04 is located 132 bytes inside of 256-byte region [0x611000003d80,0x611000003e80) freed by thread T0 here: #0 0x7f230a1f8a10 in free (/usr/lib/x86_64-linux-gnu/libasan.so.3+0xc1a10) #1 0x7f2309c2b86a in _talloc_free (/usr/lib/x86_64-linux-gnu/libtalloc.so.2+0x486a) previously allocated by thread T0 here: #0 0x7f230a1f8d28 in malloc (/usr/lib/x86_64-linux-gnu/libasan.so.3+0xc1d28) #1 0x7f2309c2dacd in _talloc_zero (/usr/lib/x86_64-linux-gnu/libtalloc.so.2+0x6acd) SUMMARY: AddressSanitizer: heap-use-after-free /home/osmocom-build/jenkins/workspace/osmo-gsm-tester_build-osmo-bsc/libosmocore/src/write_queue.c:65 in osmo_wqueue_bfd_cb Shadow bytes around the buggy address: 0x0c227fff8770: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa 0x0c227fff8780: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa 0x0c227fff8790: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa 0x0c227fff87a0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa 0x0c227fff87b0: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd =>0x0c227fff87c0:[fd]fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd 0x0c227fff87d0: fa fa fa fa fa fa fa fa fd fd fd fd fd fd fd fd 0x0c227fff87e0: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd 0x0c227fff87f0: fd fd fd fd fd fd fd fd fa fa fa fa fa fa fa fa 0x0c227fff8800: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd 0x0c227fff8810: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd Shadow byte legend (one shadow byte represents 8 application bytes): Addressable: 00 Partially addressable: 01 02 03 04 05 06 07 Heap left redzone: fa Heap right redzone: fb Freed heap region: fd Stack left redzone: f1 Stack mid redzone: f2 Stack right redzone: f3 Stack partial redzone: f4 Stack after return: f5 Stack use after scope: f8 Global redzone: f9 Global init order: f6 Poisoned by user: f7 Container overflow: fc Array cookie: ac Intra object redzone: bb ASan internal: fe Left alloca redzone: ca Right alloca redzone: cb ==12301==ABORTING
Files
Related issues
Updated by pespin almost 6 years ago
Source code line triggering the report:
fd->when &= ~BSC_FD_WRITE;
It's the first time in the function were param "struct osmo_fd* fd" is used, which probably means we are freeing the "struct osmo_fd" somewhere without unregistering the fd from the wqueue.
Updated by pespin almost 6 years ago
The ctrl interface uses a wqueue. I think it's related to how the event happen in this case:
[0;m20180424135406115 [1;32mDLCTRL[0;m <0018> control_if.c:506 accept()ed new CTRL connection from (r=10.42.42.1:53910<->l=10.42.42.7:4249) [0;m20180424135406116 [1;34mDLCTRL[0;m <0018> control_cmd.c:378 Command: GET bts.0.oml-connection-state [0;m20180424135406117 [1;34mDLINP[0;m <0013> bts_ipaccess_nanobts.c:417 Identified BTS 1/0/0 [0;m[1;36m20180424135406118 [1;34mDNM[0;m[1;36m <0005> abis_nm.c:1628 Get Attr (bts=0) [0;m[1;36m20180424135406118 [1;34mDNM[0;m[1;36m <0005> abis_nm.c:1628 Get Attr (bts=0) [0;m20180424135406118 [1;34mDCTRL[0;m <000e> osmo_bsc_ctrl.c:158 BTS connection (re)established, sending TRAP. [0;m20180424135406119 [1;32mDLCTRL[0;m <0018> control_if.c:173 close()d CTRL connection (r=10.42.42.1:53910<->l=10.42.42.7:4249)
It seems we are sending a TRAP in between the time we already created the CTRL conn and we are closing it.
Updated by pespin almost 6 years ago
My first comment is actually not true. the struct osmo_fd is actually used in the parent function in the stack (osmo_fd_disp_fds) without libasan reporting any issue. This means the osmo_fd is freed by one of the previous callbacks in osmo_wqueue_bfd_cb before accessing again the osmo_fd struct:
if (what & BSC_FD_READ) { rc = queue->read_cb(fd); if (rc == -EBADF) goto err_badfd; } if (what & BSC_FD_EXCEPT) { rc = queue->except_cb(fd); if (rc == -EBADF) goto err_badfd; }
The callbacks should imho return a specific err code to specify that the fd was released, and then the wqueue should stop using and forwarding it up the stack again.
Updated by pespin almost 6 years ago
- Status changed from New to Feedback
- % Done changed from 0 to 90
Should be fixed by https://gerrit.osmocom.org/#/c/8031/
Updated by pespin almost 6 years ago
- Status changed from Feedback to Resolved
- % Done changed from 90 to 100
Merged, closing.
Updated by pespin almost 6 years ago
- Related to Bug #3300: Avoid heap-use-after-free in osmo_wqueue_bfd_cb added