Cellular Network Infrastructure » Radio Access Network » OsmoBTS

The question is why does it stop. The system is clearly not designed that way, and I actually believe it was behaving differently in the past. You could stop and re-start the PCU as you pleased.

The PCU/PDCH message queue is separate from the other message queues to/from the DSP specifically to separate the processes. So if the PCU stops, the PDCH message queue will stall, but it should not affect the operation of the circuit switched side.

So the question is: Are we doing something wrong from osmo-bts/osmo-pcu, or is L1 not handling this correctly [anymore]?

Since 2011, osmo-bts exits when the DSP L1 is dead:
https://git.osmocom.org/osmo-bts/commit/?id=39eadbbb17c8e634b2f641c2f6148f27a975f7de

+/* we regularly check if the DSP L1 is still sending us primitives.
+ * if not, we simply stop the BTS program (and be re-spawned) */
+static void alive_timer_cb(void *data)
+{
+    struct femtol1_hdl *fl1h = data;
+
+    if (fl1h->alive_prim_cnt == 0) {
+        LOGP(DL1C, LOGL_FATAL, "DSP L1 is no longer sending primitives!\n");
+        exit(23);
+    }
+    fl1h->alive_prim_cnt = 0;
+    osmo_timer_schedule(&fl1h->alive_timer, 5, 0);
+}

So, the idea is to be re-spawned presumably by systemctl.

The question remains whether the DSP L1 dropped dead before when the PCU disconnected.
Also, a manual re-spawn didn't help, so this seems to be "new" behavior?

Verified that this issue also happens on newest sysmobts built with nightly SDK v1.5.4.
Pending: check with old sysmobts software

wirelesss, please go on and:

• reproduce this issue on your sysmoBTS.
• test a selection of older DSP firmware versions on your sysmoBTS to see whether
  older DSP firmwares show the issue or not.
• if not, find out which DSP firmware version is the first that fails this way.

I have just added a chapter to the sysmoBTS manual that describes how to exchange
firmware versions. Grab the newest manual and see section 9.6.
https://downloads.sysmocom.de/generic/documents/sysmobts-1002-manual.pdf

Log file from BTS related to the above described Bug is attached. This issue was detected and reproduced using SDK 1.5.4 with sysmoBTS. After sending command 'kill -2' to PCU, which was followed by restart of PCU and BTS processes, did not bring BTS back in operation.

wirelesss wrote:

This issue was detected and reproduced using SDK 1.5.4

The interesting information here would be the DSP/FPGA firmware version, not the SDK version.
Please describe in more detail.
Also update the Status and possibly the %Done. So far these don't match the situation at all.

Issue is detected in DSP/FPGA v3.11. Behavior is identical with the one written in the initial ticket description. Stopping of PCU process breaks logging in BTS and PCU and RBS output looks as below:

 <0007> l1sap.c:966 RACH for packet access
 <0005> paging.c:254 Add IMM.ASS to queue (group=0)
 <0007> l1sap.c:1160 deactivating channel chan_nr=0x0f trx=0
 <0007> l1_if.c:165 Tx L1 prim MPH-DEACTIVATE.req
 <0007> l1_if.c:165 Tx L1 prim MPH-DEACTIVATE.req
 <0007> l1_if.c:165 Tx L1 prim MPH-DEACTIVATE.req
 <0007> l1_if.c:165 Tx L1 prim MPH-DEACTIVATE.req
 <0007> l1sap.c:475 deactivate confirm chan_nr=0x0f trx=0
 <0000> rsl.c:517 (bts=0,trx=0,ts=7,ss=0) not sending REL ACK
 <0006> oml.c:900 DSP L1 is no longer sending primitives!
 root@sysmobts-v2:~#
 

Attempt to start PCU displays following PCUs output:

 <0001> osmobts_sock.cpp:227 Opening OsmoPCU L1 interface to OsmoBTS
 <0001> osmobts_sock.cpp:264 Failed to connect to the osmo-bts PCU socket, delaying... '/tmp/pcu_bts'
 <0001> osmobts_sock.cpp:227 Opening OsmoPCU L1 interface to OsmoBTS
 <0001> osmobts_sock.cpp:264 Failed to connect to the osmo-bts PCU socket, delaying... '/tmp/pcu_bts'
 <0001> osmobts_sock.cpp:227 Opening OsmoPCU L1 interface to OsmoBTS
 

Starting sysmobts by entering command osmo-bts-sysmo leads to below output.

root@sysmobts-v2:~# osmo-bts-sysmo -s -c /etc/osmocom/osmo-bts.cfg -M
((*))
 |
 / \ OsmoBTS
<0017> control_if.c:693 CTRL at 127.0.0.1 4238
<0010> telnet_interface.c:95 telnet at 127.0.0.1 4241
<0012> input/ipaccess.c:876 enabling ipaccess BTS mode
<0006> l1_if.c:1602 Read clock calibration(677) from EEPROM.
<0012> input/ipa.c:129 connection done.
<0012> input/ipaccess.c:700 received ID get
<0001> oml.c:217 OC=SITE-MANAGER INST=(ff,ff,ff) Tx STATE CHG REP
<0001> oml.c:217 OC=BTS INST=(00,ff,ff) Tx STATE CHG REP
<0001> oml.c:217 OC=GPRS-NSE INST=(00,ff,ff) Tx STATE CHG REP
<0001> oml.c:217 OC=GPRS-CELL INST=(00,ff,ff) Tx STATE CHG REP
<0001> oml.c:217 OC=GPRS-NSVC INST=(00,00,ff) Tx STATE CHG REP
<0001> oml.c:217 OC=GPRS-NSVC INST=(00,01,ff) Tx STATE CHG REP
<0001> oml.c:217 OC=RADIO-CARRIER INST=(00,00,ff) Tx STATE CHG REP
<0001> oml.c:217 OC=BASEBAND-TRANSCEIVER INST=(00,00,ff) Tx STATE CHG REP
<0001> oml.c:217 OC=CHANNEL INST=(00,00,00) Tx STATE CHG REP
<0001> oml.c:217 OC=CHANNEL INST=(00,00,01) Tx STATE CHG REP
<0001> oml.c:217 OC=CHANNEL INST=(00,00,02) Tx STATE CHG REP
<0001> oml.c:217 OC=CHANNEL INST=(00,00,03) Tx STATE CHG REP
<0001> oml.c:217 OC=CHANNEL INST=(00,00,04) Tx STATE CHG REP
<0001> oml.c:217 OC=CHANNEL INST=(00,00,05) Tx STATE CHG REP
<0001> oml.c:217 OC=CHANNEL INST=(00,00,06) Tx STATE CHG REP
<0001> oml.c:217 OC=CHANNEL INST=(00,00,07) Tx STATE CHG REP
<0001> oml.c:847 OC=SITE-MANAGER(00) INST=(ff,ff,ff) Rx OPSTART
<0001> oml.c:858 ... automatic ACK, OP state already was Enabled
<0001> oml.c:465 OC=BTS(01) INST=(00,ff,ff) Rx SET BTS ATTR
<0001> oml.c:547 Ignoring T200[0] (150 ms) as sent by BSC due to suspected LAPDm bug!
<0001> oml.c:547 Ignoring T200[1] (180 ms) as sent by BSC due to suspected LAPDm bug!
<0001> oml.c:547 Ignoring T200[2] (180 ms) as sent by BSC due to suspected LAPDm bug!
<0001> oml.c:547 Ignoring T200[3] (1680 ms) as sent by BSC due to suspected LAPDm bug!
<0001> oml.c:547 Ignoring T200[4] (520 ms) as sent by BSC due to suspected LAPDm bug!
<0001> oml.c:547 Ignoring T200[5] (165 ms) as sent by BSC due to suspected LAPDm bug!
<0001> oml.c:547 Ignoring T200[6] (1680 ms) as sent by BSC due to suspected LAPDm bug!
<0001> oml.c:346 Sending FOM ACK.
<0001> oml.c:875 OC=BTS(01) INST=(00,ff,ff) Rx CHG ADM STATE
<0001> oml.c:847 OC=BTS(01) INST=(00,ff,ff) Rx OPSTART
<0001> oml.c:256 OC=BTS INST=(00,ff,ff) OPER STATE NULL -> Enabled
<0001> oml.c:217 OC=BTS INST=(00,ff,ff) Tx STATE CHG REP
<0001> oml.c:249 OC=BTS INST=(00,ff,ff) AVAIL STATE Dependency -> OK
<0001> oml.c:217 OC=BTS INST=(00,ff,ff) Tx STATE CHG REP
<0001> oml.c:249 OC=GPRS-NSE INST=(00,ff,ff) AVAIL STATE Dependency -> OK
<0001> oml.c:217 OC=GPRS-NSE INST=(00,ff,ff) Tx STATE CHG REP
<0001> oml.c:249 OC=GPRS-CELL INST=(00,ff,ff) AVAIL STATE Dependency -> OK
<0001> oml.c:217 OC=GPRS-CELL INST=(00,ff,ff) Tx STATE CHG REP
<0001> oml.c:249 OC=GPRS-NSVC INST=(00,00,ff) AVAIL STATE Dependency -> OK
<0001> oml.c:217 OC=GPRS-NSVC INST=(00,00,ff) Tx STATE CHG REP
<0001> oml.c:1203 OC=GPRS-NSE(f0) INST=(00,ff,ff) Rx IPACCESS(0xf5): <0001> oml.c:1117 OC=GPRS-NSE(f0) INST=(00,ff,ff) Rx IPA SET ATTR
<0001> oml.c:346 Sending FOM ACK.
<0001> oml.c:847 OC=GPRS-NSE(f0) INST=(00,ff,ff) Rx OPSTART
<0001> oml.c:256 OC=GPRS-NSE INST=(00,ff,ff) OPER STATE NULL -> Enabled
<0001> oml.c:217 OC=GPRS-NSE INST=(00,ff,ff) Tx STATE CHG REP
<0001> oml.c:1203 OC=GPRS-CELL(f1) INST=(00,00,ff) Rx IPACCESS(0xf5): <0001> oml.c:1117 OC=GPRS-CELL(f1) INST=(00,00,ff) Rx IPA SET ATTR
<0001> oml.c:346 Sending FOM ACK.
<0001> oml.c:847 OC=GPRS-CELL(f1) INST=(00,00,ff) Rx OPSTART
<0001> oml.c:256 OC=GPRS-CELL INST=(00,ff,ff) OPER STATE NULL -> Enabled
<0001> oml.c:217 OC=GPRS-CELL INST=(00,ff,ff) Tx STATE CHG REP
<0001> oml.c:875 OC=GPRS-CELL(f1) INST=(00,00,ff) Rx CHG ADM STATE
<0001> oml.c:875 OC=GPRS-NSE(f0) INST=(00,ff,ff) Rx CHG ADM STATE
<0001> oml.c:1203 OC=GPRS-NSVC(f2) INST=(00,00,ff) Rx IPACCESS(0xf5): <0001> oml.c:1117 OC=GPRS-NSVC(f2) INST=(00,00,ff) Rx IPA SET ATTR

Command systemctl status sysmobts displays active status of BTS. A few moments later BTS status is deactivating and then activating (auto-restart) which is followed by active state. At this moment BTS returns to the operation and is capable to handle CS and PS traffic.

root@sysmobts-v2:~# systemctl status sysmobts
 sysmobts.service - sysmocom sysmoBTS
    Loaded: loaded (/lib/systemd/system/sysmobts.service; disabled)
    Active: active (running) since Thu 2016-11-17 16:09:31 UTC; 17s ago
    Process: 1030 ExecStopPost=/bin/sh -c cat /lib/firmware/sysmobts-v?.bit > /dev/fpgadl_par0 ; sleep 3s; cat /lib/firmware/sysmobts-v?.out > /dev/dspdl_dm644x_0; sleep 1s (code=exited,    status=0/SUCCESS)
  Process: 1024 ExecStopPost=/bin/sh -c echo 0 > /sys/class/leds/activity_led/brightness (code=exited, status=0/SUCCESS)
  Process: 1069 ExecStartPre=/bin/sh -c echo 0 > /sys/class/leds/activity_led/brightness (code=exited, status=0/SUCCESS)
 Main PID: 1074 (osmo-bts-sysmo)
   CGroup: /system.slice/sysmobts.service
           └─1074 /usr/bin/osmo-bts-sysmo -s -c /etc/osmocom/osmo-bts.cfg -M

root@sysmobts-v2:~# systemctl status sysmobts
   sysmobts.service - sysmocom sysmoBTS
   Loaded: loaded (/lib/systemd/system/sysmobts.service; disabled)
   Active: deactivating (stop-post) (Result: exit-code) since Thu 2016-11-17 16:10:02 UTC; 2s ago
   Process: 1078 ExecStopPost=/bin/sh -c echo 0 > /sys/class/leds/activity_led/brightness (code=exited, status=0/SUCCESS)
   Process: 1074 ExecStart=/usr/bin/osmo-bts-sysmo -s -c /etc/osmocom/osmo-bts.cfg -M (code=exited, status=23)
   Process: 1069 ExecStartPre=/bin/sh -c echo 0 > /sys/class/leds/activity_led/brightness (code=exited, status=0/SUCCESS)
  Main PID: 1074 (code=exited, status=23);         : 1084 (sh)
    CGroup: /system.slice/sysmobts.service
           └─control
             ├─1084 /bin/sh -c cat /lib/firmware/sysmobts-v?.bit > /dev/fpgadl_par0 ; sleep 3s; cat /lib/firmware/sysmobts-v?.out > /dev/dspdl_dm644x_0; sleep 1s
             └─1095 sleep 3s

root@sysmobts-v2:~# systemctl status sysmobts
 sysmobts.service - sysmocom sysmoBTS
   Loaded: loaded (/lib/systemd/system/sysmobts.service; disabled)
   Active: deactivating (stop-post) (Result: exit-code) since Thu 2016-11-17 16:10:02 UTC; 4s ago
  Process: 1078 ExecStopPost=/bin/sh -c echo 0 > /sys/class/leds/activity_led/brightness (code=exited, status=0/SUCCESS)
  Process: 1074 ExecStart=/usr/bin/osmo-bts-sysmo -s -c /etc/osmocom/osmo-bts.cfg -M (code=exited, status=23)
  Process: 1069 ExecStartPre=/bin/sh -c echo 0 > /sys/class/leds/activity_led/brightness (code=exited, status=0/SUCCESS)
  Main PID: 1074 (code=exited, status=23);         : 1084 (sh)
   CGroup: /system.slice/sysmobts.service
           └─control
             ├─1084 /bin/sh -c cat /lib/firmware/sysmobts-v?.bit > /dev/fpgadl_par0 ; sleep 3s; cat /lib/firmware/sysmobts-v?.out > /dev/dspdl_dm644x_0; sleep 1s
             └─1099 sleep 1s

root@sysmobts-v2:~# systemctl status sysmobts
 sysmobts.service - sysmocom sysmoBTS
   Loaded: loaded (/lib/systemd/system/sysmobts.service; disabled)
   Active: activating (auto-restart) (Result: exit-code) since Thu 2016-11-17 16:10:07 UTC; 100ms ago
   Process: 1084 ExecStopPost=/bin/sh -c cat /lib/firmware/sysmobts-v?.bit > /dev/fpgadl_par0 ; sleep 3s; cat /lib/firmware/sysmobts-v?.out > /dev/dspdl_dm644x_0; sleep 1s (code=exited,  status=0/SUCCESS)
   Process: 1078 ExecStopPost=/bin/sh -c echo 0 > /sys/class/leds/activity_led/brightness (code=exited, status=0/SUCCESS)
   Process: 1074 ExecStart=/usr/bin/osmo-bts-sysmo -s -c /etc/osmocom/osmo-bts.cfg -M (code=exited, status=23)
   Process: 1069 ExecStartPre=/bin/sh -c echo 0 > /sys/class/leds/activity_led/brightness (code=exited, status=0/SUCCESS)
  Main PID: 1074 (code=exited, status=23)

root@sysmobts-v2:~# systemctl status sysmobts
 sysmobts.service - sysmocom sysmoBTS
    Loaded: loaded (/lib/systemd/system/sysmobts.service; disabled)
    Active: activating (auto-restart) (Result: exit-code) since Thu 2016-11-17 16:10:07 UTC; 1s ago
    Process: 1084 ExecStopPost=/bin/sh -c cat /lib/firmware/sysmobts-v?.bit > /dev/fpgadl_par0 ; sleep 3s; cat /lib/firmware/sysmobts-v?.out > /dev/dspdl_dm644x_0; sleep 1s (code=exited,  status=0/SUCCESS)
   Process: 1078 ExecStopPost=/bin/sh -c echo 0 > /sys/class/leds/activity_led/brightness (code=exited, status=0/SUCCESS)
   Process: 1074 ExecStart=/usr/bin/osmo-bts-sysmo -s -c /etc/osmocom/osmo-bts.cfg -M (code=exited, status=23)
   Process: 1069 ExecStartPre=/bin/sh -c echo 0 > /sys/class/leds/activity_led/brightness (code=exited, status=0/SUCCESS)
  Main PID: 1074 (code=exited, status=23)

root@sysmobts-v2:~# systemctl status sysmobts
 sysmobts.service - sysmocom sysmoBTS
    Loaded: loaded (/lib/systemd/system/sysmobts.service; disabled)
    Active: active (running) since Thu 2016-11-17 16:10:09 UTC; 1s ago
   Process: 1084 ExecStopPost=/bin/sh -c cat /lib/firmware/sysmobts-v?.bit > /dev/fpgadl_par0 ; sleep 3s; cat /lib/firmware/sysmobts-v?.out > /dev/dspdl_dm644x_0; sleep 1s (code=exited, status=0/SUCCESS)
   Process: 1078 ExecStopPost=/bin/sh -c echo 0 > /sys/class/leds/activity_led/brightness (code=exited, status=0/SUCCESS)
   Process: 1104 ExecStartPre=/bin/sh -c echo 0 > /sys/class/leds/activity_led/brightness (code=exited, status=0/SUCCESS)
  Main PID: 1108 (osmo-bts-sysmo)
    CGroup: /system.slice/sysmobts.service
           └─1108 /usr/bin/osmo-bts-sysmo -s -c /etc/osmocom/osmo-bts.cfg -M

root@sysmobts-v2:~# systemctl status sysmobts
 sysmobts.service - sysmocom sysmoBTS
    Loaded: loaded (/lib/systemd/system/sysmobts.service; disabled)
    Active: active (running) since Thu 2016-11-17 16:10:09 UTC; 2s ago
   Process: 1084 ExecStopPost=/bin/sh -c cat /lib/firmware/sysmobts-v?.bit > /dev/fpgadl_par0 ; sleep 3s; cat /lib/firmware/sysmobts-v?.out > /dev/dspdl_dm644x_0; sleep 1s (code=exited, status=0/SUCCESS)
   Process: 1078 ExecStopPost=/bin/sh -c echo 0 > /sys/class/leds/activity_led/brightness (code=exited, status=0/SUCCESS)
   Process: 1104 ExecStartPre=/bin/sh -c echo 0 > /sys/class/leds/activity_led/brightness (code=exited, status=0/SUCCESS)
  Main PID: 1108 (osmo-bts-sysmo)
    CGroup: /system.slice/sysmobts.service
           └─1108 /usr/bin/osmo-bts-sysmo -s -c /etc/osmocom/osmo-bts.cfg -M

Some additional comments to the previous post.

Firmware v3.11
------------------------
When the PCU is stopped by issuing of 'kill -2' command, sysmobts.service goes to status disabled. Starting sysmobts.service process brings RF cell up.
In any case stopping of PCU provokes disabling of sysmobts.service and it is detected in this version of Firmware.

Firmware v3.1
---------------------
When using firmware v3.1, osmo-bts-sysmo attempt to start automatically but after a while failed. Issuing command /usr/bin/osmo-bts-sysmo -s -c /etc/osmocom/osmo-bts.cfg -M
shows :

<0001> oml.c:346 Sending FOM ACK.
 <0001> oml.c:847 OC=GPRS-NSE(f0) INST=(00,ff,ff) Rx OPSTART
 <0001> oml.c:256 OC=GPRS-NSE INST=(00,ff,ff) OPER STATE NULL -> Enabled
 <0001> oml.c:217 OC=GPRS-NSE INST=(00,ff,ff) Tx STATE CHG REP
 <0001> oml.c:1203 OC=GPRS-CELL(f1) INST=(00,00,ff) Rx IPACCESS(0xf5): <0001> oml.c:1117 OC=GPRS-CELL(f1) INST=(00,00,ff) Rx IPA SET ATTR
 <0001> oml.c:346 Sending FOM ACK.
 <0001> oml.c:847 OC=GPRS-CELL(f1) INST=(00,00,ff) Rx OPSTART
 <0001> oml.c:256 OC=GPRS-CELL INST=(00,ff,ff) OPER STATE NULL -> Enabled
 <0001> oml.c:217 OC=GPRS-CELL INST=(00,ff,ff) Tx STATE CHG REP
 <0001> oml.c:875 OC=GPRS-CELL(f1) INST=(00,00,ff) Rx CHG ADM STATE
 <0001> oml.c:875 OC=GPRS-NSE(f0) INST=(00,ff,ff) Rx CHG ADM STATE
 <0001> oml.c:1203 OC=GPRS-NSVC(f2) INST=(00,00,ff) Rx IPACCESS(0xf5): <0001> oml.c:1117 OC=GPRS-NSVC(f2) INST=(00,00,ff) Rx IPA SET ATTR
 <0001> oml.c:346 Sending FOM ACK.
 <0001> oml.c:847 OC=GPRS-NSVC(f2) INST=(00,00,ff) Rx OPSTART
 <0001> oml.c:256 OC=GPRS-NSVC INST=(00,00,ff) OPER STATE NULL -> Enabled
 <0001> oml.c:217 OC=GPRS-NSVC INST=(00,00,ff) Tx STATE CHG REP
 <0001> oml.c:875 OC=GPRS-NSVC(f2) INST=(00,00,ff) Rx CHG ADM STATE 
 <0001> oml.c:465 OC=BTS(01) INST=(00,ff,ff) Rx SET BTS ATTR
 <0001> oml.c:547 Ignoring T200[0] (150 ms) as sent by BSC due to suspected LAPDm bug!
 <0001> oml.c:547 Ignoring T200[1] (180 ms) as sent by BSC due to suspected LAPDm bug!
 <0001> oml.c:547 Ignoring T200[2] (180 ms) as sent by BSC due to suspected LAPDm bug!
 <0001> oml.c:547 Ignoring T200[3] (1680 ms) as sent by BSC due to suspected LAPDm bug!
 <0001> oml.c:547 Ignoring T200[4] (520 ms) as sent by BSC due to suspected LAPDm bug!
 <0001> oml.c:547 Ignoring T200[5] (165 ms) as sent by BSC due to suspected LAPDm bug!
 <0001> oml.c:547 Ignoring T200[6] (1680 ms) as sent by BSC due to suspected LAPDm bug!
 <0001> oml.c:346 Sending FOM ACK.
 <0001> oml.c:875 OC=BTS(01) INST=(00,ff,ff) Rx CHG ADM STATE
 <0001> oml.c:901 ADM state already was Unlocked
 <0001> oml.c:847 OC=BTS(01) INST=(00,ff,ff) Rx OPSTART
 <0001> oml.c:858 ... automatic ACK, OP state already was Enabled
 <0001> oml.c:847 OC=RADIO-CARRIER(02) INST=(00,00,ff) Rx OPSTART
 <0001> oml.c:614 OC=RADIO-CARRIER(02) INST=(00,00,ff) Rx SET RADIO CARRIER ATTR
 <0001> oml.c:642 Set RF Max Power Reduction = 20 dBm
 <0001> oml.c:346 Sending FOM ACK.
 <0001> oml.c:875 OC=RADIO-CARRIER(02) INST=(00,00,ff) Rx CHG ADM STATE
 <0001> oml.c:901 ADM state already was Unlocked
 <0006> l1_if.c:1394 RF-MUTE.req not supported by SuperFemto
 <0012> input/ipa.c:67 connection closed with server
 <000d> abis.c:135 Signalling link down
 <0001> bts.c:206 Shutting down BTS 0, Reason Abis close
 <0006> oml.c:450 Close TRX 0
 <0007> l1_if.c:165 Tx L1 prim MPH-CLOSE.req
 <0001> oml.c:249 OC=RADIO-CARRIER INST=(00,00,ff) AVAIL STATE OK -> Off line
 <0001> oml.c:256 OC=RADIO-CARRIER INST=(00,00,ff) OPER STATE Disabled -> Disabled
 <0001> oml.c:217 OC=RADIO-CARRIER INST=(00,00,ff) Tx STATE CHG REP
 <0001> oml.c:249 OC=BASEBAND-TRANSCEIVER INST=(00,00,ff) AVAIL STATE OK -> Off line
 <0001> oml.c:256 OC=BASEBAND-TRANSCEIVER INST=(00,00,ff) OPER STATE NULL -> Disabled
 <0001> oml.c:217 OC=BASEBAND-TRANSCEIVER INST=(00,00,ff) Tx STATE CHG REP
 Shutdown timer expired
 root@sysmobts-v2:~#

Firmware v2.7
---------------------
During cross-compiling procedure following error message appears:

osmo-bts-sysmo/sysmo_l1_if.c:340:18: note: each undeclared identifier is reported only once for each function it appears in
 Makefile:642: recipe for target 'osmo-bts-sysmo/sysmo_l1_if.o' failed
 make[1]:*** [osmo-bts-sysmo/sysmo_l1_if.o] Error 1
 make[1]:*** Waiting for unfinished jobs....
 In file included from osmo-bts-sysmo/sysmo_l1_hw.c:45:0:
 osmo-bts-sysmo/femtobts.h:25:46: error: 'SuperFemto_PrimId_NUM' undeclared here (not in a function)
 const enum l1prim_type femtobts_sysprim_type[SuperFemto_PrimId_NUM];
                                              ^
 osmo-bts-sysmo/femtobts.h:27:1: error: unknown type name 'SuperFemto_PrimId_t'
 const SuperFemto_PrimId_t femtobts_sysprim_req2conf[SuperFemto_PrimId_NUM];
 ^

Firmware 3.11:
Killing PCU by issuing of 'kill -2' command disables sysmobts.service. It is possible to start sysmobts.service manually this brings RF cell on air.

Firmware v3.1:
osmo-bts-sysmo can not start after killing pcu.

Firmware v2.7:
Cross-compiling procedure fails with error message, which is shown in the above post.

wirelesss wrote:

Firmware 3.11:
Killing PCU by issuing of 'kill -2' command disables sysmobts.service. It is possible to start sysmobts.service manually this brings RF cell on air.

Firmware 3.10:
After change of Firmware to 3.10 followed by powercycle of sysmoBTS leads to cycling restarts of sysmopcu.service.
systemctl command shows stable active (running) state of sysmobts.service.

Firmware v3.1:
osmo-bts-sysmo can not start after killing pcu.

Firmware v2.7:
Cross-compiling procedure fails with error message, which is shown in the above post.

wirelesss wrote:

wirelesss wrote:

Firmware 3.11:
Killing PCU by issuing of 'kill -2' command disables sysmobts.service. It is possible to start sysmobts.service manually this brings RF cell on air.

Firmware 3.10:
After change of firmware to 3.10 followed by power-cycle of sysmoBTS leads to cycling restarts of sysmopcu.service.
systemctl command shows stable active (running) state of sysmobts.service. The LED indicator "RF state" is off.

Firmware 3.6:
Changing firmware to 3.6 and power-cycling the sysmoBTS leads to cycling restarts of sysmopcu.service.
sysmobts.service is in stable active (running) state. The LED indicator "RF state" is off.

Firmware v3.1:
osmo-bts-sysmo can not start after killing pcu.

Firmware v2.7:
Cross-compiling procedure fails with error message, which is shown in the above post.

Once this happened, another way to make the BTS usable is to re-flash the DSP images:

cat /lib/firmware/sysmobts-v?.bit > /dev/fpgadl_par0 ; sleep 3s; cat /lib/firmware/sysmobts-v?.out > /dev/dspdl_dm644x_0; sleep 1s

The firwmare v3.1 log output looks like the link to the osmo-nitb went down,
so I assume there was a problem unrelated to this question whether the L1 communication breaks down.
We should re-test v3.1 and make sure that it wasn't an odd osmo-nitb problem.

Otherwise I'm not sure what to do here. It seems to take substantial effort to get things
to run with older firmwares (3.10, 3.6, 2.7).

So far we have no conclusion that would allow requesting Nutaq to fix a regression, yet.
We could inform Nutaq of the problem without a reference to an older working state...?

v 3.1 of the firmware has been re-tested.

osmo-bts-sysmo and osmo-pcu make cycling restarts. The RF Active link comes for a few seconds and then goes off.

A log from bts is shown below. I have also take pcaps from eth and lo interfaces.

/usr/bin/osmo-bts-sysmo -s -c /etc/osmocom/osmo-bts.cfg -M

((*))
  |
 / \ OsmoBTS
<0017> control_if.c:693 CTRL at 127.0.0.1 4238
<0010> telnet_interface.c:95 telnet at 127.0.0.1 4241
<0012> input/ipaccess.c:885 enabling ipaccess BTS mode, OML connecting to 10.9.1.112:3002
<0006> phy_link.c:58 PHY link state change shutdown -> connectiong
<0006> l1_if.c:1622 sysmoBTSv2 L1IF compiled against API headers v3.1.0
<0006> l1_if.c:1610 Read clock calibration(677) from EEPROM.
<0006> l1_if.c:183 Tx SYS prim LAYER1-RESET.req
<0006> phy_link.c:58 PHY link state change connectiong -> connected
<0006> phy_link.c:68 trx_set_avail(1)
<0012> input/ipa.c:129 connection done.
<0012> input/ipaccess.c:706 received ID get
<0001> oml.c:156 OC=SITE-MANAGER INST=(ff,ff,ff) Tx STATE CHG REP
<0001> oml.c:156 OC=BTS INST=(00,ff,ff) Tx STATE CHG REP
<0001> oml.c:156 OC=GPRS-NSE INST=(00,ff,ff) Tx STATE CHG REP
<0001> oml.c:156 OC=GPRS-CELL INST=(00,ff,ff) Tx STATE CHG REP
<0001> oml.c:156 OC=GPRS-NSVC INST=(00,00,ff) Tx STATE CHG REP
<0001> oml.c:156 OC=GPRS-NSVC INST=(00,01,ff) Tx STATE CHG REP
<0001> oml.c:156 OC=RADIO-CARRIER INST=(00,00,ff) Tx STATE CHG REP
<0001> oml.c:156 OC=BASEBAND-TRANSCEIVER INST=(00,00,ff) Tx STATE CHG REP
<0001> oml.c:156 OC=CHANNEL INST=(00,00,00) Tx STATE CHG REP
<0001> oml.c:156 OC=CHANNEL INST=(00,00,01) Tx STATE CHG REP
<0001> oml.c:156 OC=CHANNEL INST=(00,00,02) Tx STATE CHG REP
<0001> oml.c:156 OC=CHANNEL INST=(00,00,03) Tx STATE CHG REP
<0001> oml.c:156 OC=CHANNEL INST=(00,00,04) Tx STATE CHG REP
<0001> oml.c:156 OC=CHANNEL INST=(00,00,05) Tx STATE CHG REP
<0001> oml.c:156 OC=CHANNEL INST=(00,00,06) Tx STATE CHG REP
<0001> oml.c:156 OC=CHANNEL INST=(00,00,07) Tx STATE CHG REP
<0006> l1_if.c:1496 Rx L1-RESET.conf (status=Success)
<0006> l1_if.c:1533 Tx SET-TRACE-FLAGS.req (0x00000000)
<0006> l1_if.c:183 Tx SYS prim SYSTEM-INFO.req
<0006> l1_if.c:1440 DSP v3.1.0, FPGA v3.1.0
n<0006> l1_if.c:1309 Using external attenuator.
<0006> l1_if.c:183 Tx SYS prim ACTIVATE-RF.req
<0006> calib_file.c:191 No calibration table fix-up needed, firmware < 3.3
<0006> l1_if.c:183 Tx SYS prim SET-RX-CALIB.req
<0001> oml.c:473 Ignoring T200[0] (150 ms) as sent by BSC due to suspected LAPDm bug!
<0001> oml.c:473 Ignoring T200[1] (180 ms) as sent by BSC due to suspected LAPDm bug!
<0001> oml.c:473 Ignoring T200[2] (180 ms) as sent by BSC due to suspected LAPDm bug!
<0001> oml.c:473 Ignoring T200[3] (1680 ms) as sent by BSC due to suspected LAPDm bug!
<0001> oml.c:473 Ignoring T200[4] (520 ms) as sent by BSC due to suspected LAPDm bug!
<0001> oml.c:473 Ignoring T200[5] (165 ms) as sent by BSC due to suspected LAPDm bug!
<0001> oml.c:473 Ignoring T200[6] (1680 ms) as sent by BSC due to suspected LAPDm bug!
<0001> oml.c:195 OC=BTS INST=(00,ff,ff) OPER STATE NULL -> Enabled
<0001> oml.c:156 OC=BTS INST=(00,ff,ff) Tx STATE CHG REP
<0001> oml.c:188 OC=BTS INST=(00,ff,ff) AVAIL STATE Dependency -> OK
<0001> oml.c:156 OC=BTS INST=(00,ff,ff) Tx STATE CHG REP
<0001> oml.c:188 OC=GPRS-NSE INST=(00,ff,ff) AVAIL STATE Dependency -> OK
<0001> oml.c:156 OC=GPRS-NSE INST=(00,ff,ff) Tx STATE CHG REP
<0001> oml.c:188 OC=GPRS-CELL INST=(00,ff,ff) AVAIL STATE Dependency -> OK
<0001> oml.c:156 OC=GPRS-CELL INST=(00,ff,ff) Tx STATE CHG REP
<0001> oml.c:188 OC=GPRS-NSVC INST=(00,00,ff) AVAIL STATE Dependency -> OK
<0001> oml.c:156 OC=GPRS-NSVC INST=(00,00,ff) Tx STATE CHG REP
<0006> l1_if.c:1213 Rx RF-ACT.conf (status=Success)
<0006> main.c:109 Set global status #0 to 1 (0000 -> 0001), LEDs: ACT 1
<0001> oml.c:188 OC=RADIO-CARRIER INST=(00,00,ff) AVAIL STATE Power off -> OK
<0001> oml.c:195 OC=RADIO-CARRIER INST=(00,00,ff) OPER STATE NULL -> Disabled
<0001> oml.c:156 OC=RADIO-CARRIER INST=(00,00,ff) Tx STATE CHG REP
<0001> oml.c:301 OC=RADIO-CARRIER INST=(00,00,ff) Tx SW ACT REP
<0001> oml.c:188 OC=BASEBAND-TRANSCEIVER INST=(00,00,ff) AVAIL STATE Power off -> OK
<0001> oml.c:156 OC=BASEBAND-TRANSCEIVER INST=(00,00,ff) Tx STATE CHG REP
<0001> oml.c:301 OC=BASEBAND-TRANSCEIVER INST=(00,00,ff) Tx SW ACT REP
<0001> oml.c:188 OC=CHANNEL INST=(00,00,00) AVAIL STATE Power off -> Dependency
<0001> oml.c:195 OC=CHANNEL INST=(00,00,00) OPER STATE NULL -> Disabled
<0001> oml.c:156 OC=CHANNEL INST=(00,00,00) Tx STATE CHG REP
<0001> oml.c:188 OC=CHANNEL INST=(00,00,01) AVAIL STATE Power off -> Dependency
<0001> oml.c:195 OC=CHANNEL INST=(00,00,01) OPER STATE NULL -> Disabled
<0001> oml.c:156 OC=CHANNEL INST=(00,00,01) Tx STATE CHG REP
<0001> oml.c:188 OC=CHANNEL INST=(00,00,02) AVAIL STATE Power off -> Dependency
<0001> oml.c:195 OC=CHANNEL INST=(00,00,02) OPER STATE NULL -> Disabled