Cellular Network Infrastructure » Radio Access Network » OsmoPCU

On May 5 2022 job 'master-osmo-pcu' failed:

https://jenkins.osmocom.org/jenkins/view/master/job/master-osmo-pcu/4825/

in particular, the 'rpi4-raspbian11' configuration:

https://jenkins.osmocom.org/jenkins/view/master/job/master-osmo-pcu/4825/FIRMWARE_VERSION=master,WITH_MANUALS=0,label=rpi4-raspbian11,with_dsp=none,with_vty=False/

as can be seen from the Console, an RPi4 slave hits a heap-use-after-free while running the 'tbf' test:

--- experr    2022-05-06 00:12:16.383377168 +0000
+++ /build/osmo-pcu-1.0.0.13-0bda/_build/sub/tests/testsuite.dir/at-groups/4/stderr    2022-05-06 00:12:24.423181086 +0000
@@ -9154,4 +9154,56 @@
 TBF(UL-TFI_-1){ASSIGN}: Deallocated
 UL_ASS_TBF(UL-TFI_-1){NONE}: Deallocated
 DL_ASS_TBF(UL-TFI_-1){NONE}: Deallocated
-=== end test_packet_access_rej_prr_no_other_tbfs ===
+TBF(TFI=1 TLLI=0xf1223344 DIR=DL STATE=FINISHED) T3191 timeout expired, freeing TBF: |Assignment was on PACCH|No downlink ACK received yet|
+=================================================================
+==19647==ERROR: AddressSanitizer: heap-use-after-free on address 0xb3e53e0c at pc 0x005d9129 bp 0xbe8fd518 sp 0xbe8fd51c
+READ of size 4 at 0xb3e53e0c thread T0
+    #0 0x5d9127 in bts_do_rate_ctr_inc (/build/osmo-pcu-1.0.0.13-0bda/_build/sub/tests/tbf/TbfTest+0x169127)
+    #1 0x5e14d7 in tbf_free (/build/osmo-pcu-1.0.0.13-0bda/_build/sub/tests/tbf/TbfTest+0x1714d7)
+    #2 0x5e68d3 in tbf_timeout_free(gprs_rlcmac_tbf*, tbf_timers, bool) (/build/osmo-pcu-1.0.0.13-0bda/_build/sub/tests/tbf/TbfTest+0x1768d3)
+    #3 0x5e693f in cb_T3191(void*) (/build/osmo-pcu-1.0.0.13-0bda/_build/sub/tests/tbf/TbfTest+0x17693f)
+
+0xb3e53e0c is located 23564 bytes inside of 23720-byte region [0xb3e4e200,0xb3e53ea8)
+freed by thread T0 here:
+    #0 0xb6a5d47d in free (/usr/lib/arm-linux-gnueabihf/libasan.so.3+0x9247d)
+
+previously allocated by thread T0 here:
+    #0 0xb6a5d69b in __interceptor_malloc (/usr/lib/arm-linux-gnueabihf/libasan.so.3+0x9269b)
+    #1 0x1d  (<unknown module>)
+    #2 0x1  (<unknown module>)
+
+SUMMARY: AddressSanitizer: heap-use-after-free (/build/osmo-pcu-1.0.0.13-0bda/_build/sub/tests/tbf/TbfTest+0x169127) in bts_do_rate_ctr_inc
+Shadow bytes around the buggy address:
+  0x367ca770: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
+  0x367ca780: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
+  0x367ca790: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
+  0x367ca7a0: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
+  0x367ca7b0: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
+=>0x367ca7c0: fd[fd]fd fd fd fd fd fd fd fd fd fd fd fd fd fd
+  0x367ca7d0: fd fd fd fd fd fa fa fa fa fa fa fa fa fa fa fa
+  0x367ca7e0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
+  0x367ca7f0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
+  0x367ca800: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
+  0x367ca810: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
+Shadow byte legend (one shadow byte represents 8 application bytes):
+  Addressable:           00
+  Partially addressable: 01 02 03 04 05 06 07 
+  Heap left redzone:       fa
+  Heap right redzone:      fb
+  Freed heap region:       fd
+  Stack left redzone:      f1
+  Stack mid redzone:       f2
+  Stack right redzone:     f3
+  Stack partial redzone:   f4
+  Stack after return:      f5
+  Stack use after scope:   f8
+  Global redzone:          f9
+  Global init order:       f6
+  Poisoned by user:        f7
+  Container overflow:      fc
+  Array cookie:            ac
+  Intra object redzone:    bb
+  ASan internal:           fe
+  Left alloca redzone:     ca
+  Right alloca redzone:    cb
+==19647==ABORTING
stdout:
../../../tests/testsuite.at:28: exit code was 1, expected 0
4. testsuite.at:25: 4. tbf (testsuite.at:25): FAILED (testsuite.at:28)

This looks like a race condition to me. Next build 4826 is back to normal.