Project

General

Profile

Actions

Bug #4089

closed

libosmovty: ASan heap-use-after-free in osmo-trx while messing with VTY (debug enabled)

Added by pespin over 4 years ago. Updated over 4 years ago.

Status:
Resolved
Priority:
Normal
Assignee:
Category:
libosmovty
Target version:
-
Start date:
07/04/2019
Due date:
% Done:

100%

Spec Reference:

Description

So I was running my osmo-trx dev branch (adding TRXDv1 there), and I opened a VTY session on an osmo-trx sending samples to the BTSTRX, enabled logging and did "logging level set-all debug". Then left it for a while with a LOT of output being printed (from several threads). Then started pressing "enter" on my terminal to get to the end and then started going up trying to catch one sepcific line. Then I noticed the terminal was not outputing more lines and saw this:

Thu Jul  4 18:01:55 2019 DMAIN <0000> Transceiver.cpp:1076 [tid=139662515480320] ClockInterface: sending IND CLOCK 880797
Thu Jul  4 18:01:56 2019 DMAIN <0000> Transceiver.cpp:1076 [tid=139662515480320] ClockInterface: sending IND CLOCK 881013
Thu Jul  4 18:01:57 2019 DMAIN <0000> Transceiver.cpp:1076 [tid=139662515480320] ClockInterface: sending IND CLOCK 881230
Thu Jul  4 18:01:58 2019 DMAIN <0000> Transceiver.cpp:1076 [tid=139662515480320] ClockInterface: sending IND CLOCK 881446
=================================================================
==16880==ERROR: AddressSanitizer: heap-use-after-free on address 0x6210007b2168 at pc 0x7f05c0af52a8 bp 0x7f05b71f6170 sp 0x7f05b71f6160
READ of size 8 at 0x6210007b2168 thread T26 (TxLower)
U    #0 0x7f05c0af52a7 in buffer_put /git/libosmocore/src/vty/buffer.c:181
    #1 0x7f05c0b15f4c in vty_out_va /git/libosmocore/src/vty/vty.c:294
    #2 0x7f05c0b161cf in vty_out /git/libosmocore/src/vty/vty.c:315
    #3 0x7f05c0b30144 in _vty_output /git/libosmocore/src/vty/logging_vty.c:85
    #4 0x7f05c08d49f4 in _output /git/libosmocore/src/logging.c:460
    #5 0x7f05c08d546a in osmo_vlogp /git/libosmocore/src/logging.c:548
    #6 0x7f05c08d58a4 in logp2 /git/libosmocore/src/logging.c:581
    #7 0x5651286501d3 in Log::~Log() /git/osmo-trx/CommonLibs/Logger.cpp:51
    #8 0x565128630c85 in Transceiver::driveTxFIFO() /git/osmo-trx/Transceiver52M/Transceiver.cpp:1032
    #9 0x565128631547 in TxLowerLoopAdapter(Transceiver*) /git/osmo-trx/Transceiver52M/Transceiver.cpp:1126
    #10 0x7f05c168ba91 in start_thread (/usr/lib/libpthread.so.0+0x7a91)
    #11 0x7f05bf74acd2 in __clone (/usr/lib/libc.so.6+0xfacd2)

0x6210007b2168 is located 104 bytes inside of 4216-byte region [0x6210007b2100,0x6210007b3178)
freed by thread T0 here:
    #0 0x7f05c1796f89 in __interceptor_free /build/gcc/src/gcc/libsanitizer/asan/asan_malloc_linux.cc:66
    #1 0x7f05c07a3143 in _talloc_free (/usr/lib/libtalloc.so.2+0x6143)

previously allocated by thread T27 (RxLower) here:
    #0 0x7f05c1797389 in __interceptor_malloc /build/gcc/src/gcc/libsanitizer/asan/asan_malloc_linux.cc:86
    #1 0x7f05c07a5e11 in _talloc_zero (/usr/lib/libtalloc.so.2+0x8e11)

Thread T26 (TxLower) created by T24 (CtrlService0) here:
    #0 0x7f05c16f76d5 in __interceptor_pthread_create /build/gcc/src/gcc/libsanitizer/asan/asan_interceptors.cc:202
    #1 0x56512864f7b6 in Thread::start(void* (*)(void*), void*) /git/osmo-trx/CommonLibs/Threads.cpp:142

Thread T24 (CtrlService0) created by T0 here:
    #0 0x7f05c16f76d5 in __interceptor_pthread_create /build/gcc/src/gcc/libsanitizer/asan/asan_interceptors.cc:202
    #1 0x56512864f7b6 in Thread::start(void* (*)(void*), void*) /git/osmo-trx/CommonLibs/Threads.cpp:142

Thread T27 (RxLower) created by T24 (CtrlService0) here:
    #0 0x7f05c16f76d5 in __interceptor_pthread_create /build/gcc/src/gcc/libsanitizer/asan/asan_interceptors.cc:202
    #1 0x56512864f7b6 in Thread::start(void* (*)(void*), void*) /git/osmo-trx/CommonLibs/Threads.cpp:142

SUMMARY: AddressSanitizer: heap-use-after-free /git/libosmocore/src/vty/buffer.c:181 in buffer_put
Shadow bytes around the buggy address:
  0x0c42800ee3d0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c42800ee3e0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c42800ee3f0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c42800ee400: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c42800ee410: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
=>0x0c42800ee420: fd fd fd fd fd fd fd fd fd fd fd fd fd[fd]fd fd
  0x0c42800ee430: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
  0x0c42800ee440: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
  0x0c42800ee450: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
  0x0c42800ee460: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
  0x0c42800ee470: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
Shadow byte legend (one shadow byte represents 8 application bytes):
  Addressable:           00
  Partially addressable: 01 02 03 04 05 06 07
  Heap left redzone:       fa
  Freed heap region:       fd
  Stack left redzone:      f1
  Stack mid redzone:       f2
  Stack right redzone:     f3
  Stack after return:      f5
  Stack use after scope:   f8
  Global redzone:          f9
  Global init order:       f6
  Poisoned by user:        f7
  Container overflow:      fc
  Array cookie:            ac
  Intra object redzone:    bb
  ASan internal:           fe
  Left alloca redzone:     ca
  Right alloca redzone:    cb
==16880==ABORTING

So it's not clear to me if the issue is related to multi-threading (see #4088) and perhaps BTSTRX dropping the connection due to clock issues with debug enabled, or simply because there's a bug in VTY code. It needs to be investigated.

Using libosmocore 316d1e1b7bbe4c6d1c9b6adbd27ecba3b20f3743.


Related issues

Related to libosmocore - Bug #4088: logging: Add API to enable mutex protecting around llogging target listResolvedpespin07/04/2019

Actions
Actions #1

Updated by pespin over 4 years ago

  • Related to Bug #4088: logging: Add API to enable mutex protecting around llogging target list added
Actions #2

Updated by pespin over 4 years ago

  • Status changed from New to Feedback

I was able to reproduce this bug one more time a few days ago.

It may become fixed by logging mutex patches from #4088. I leave it in feedback for a while after they are merged in order to see if I can still reproduce it.

Actions #3

Updated by pespin over 4 years ago

  • Status changed from Feedback to Resolved
  • % Done changed from 0 to 100

Haven't been able to reproduce it for a while, closing ticket.

Actions

Also available in: Atom PDF

Add picture from clipboard (Maximum size: 48.8 MB)