Cellular Network Infrastructure » cni-legacy » OpenBSC

While the test case without any real LAPDm instance on the MS side is of course fairly artificial, this can of course also occur in normal operation, in at least the following cases:

• MS runs out of coverage, out of power or reboots before being able to send LAPDm DISC (genreating RELEASE INDICATION at BTS)
• radio interference at the time of the LAPDm DISC
• malicious or buggy MS

This might explain some very sporadic failure cases in real-world setups

The problem was that in the specific config file I was using T3109 was set to 0. And without T3109 being ever started, the erroneous channel release procedure doesn't complete. I still think this is a bug. we either should reject a value of zero, or otherwise remain in a clean / sane state, no matter what the user configures.

See patches https://gerrit.osmocom.org/3305 and https://gerrit.osmocom.org/3306 adressing this problem.

Please note that even the osmo-gsm-tester config file templates suffer from this, see https://gerrit.osmocom.org/3308

all three related gerrit changes merged.

My personal thank you for fixing this. It's been a very annoying issue in our installations.

laforge wrote:

On Thu, Jul 20, 2017 at 10:27:16AM +0000, ipse [REDMINE] wrote:

My personal thank you for fixing this. It's been a very annoying issue in our installations.

Did you have T3109 set to 0 in your config? If not, it might be another
bug, so don't thank me too soon.

Yes, they were set to 0, because we copy-pasted that part of the config from examples. I've manually changed them to the defaults from your patch - we'll see if we see stuck channels again in the next few days.

We've monitored our installation were we changed the timer values to the reasonable defaults and amount of stuck channels seem to be better but is not 100% gone. After a surge in traffic we've got one BTS which has 2 stuck channels. BTS is still running with those channels stuck - let me know if you want me to provide some more debugging info about it.

PS I didn't restart osmo-nitb or osmo-bts's after setting timer values, as I assume they are applied immediately to new channels. Is that correct?

We've just got another stuck channel in the following state, so it looks like the issue is not completely resolved:

OpenBSC> show lchan 6 1 0 
BTS 6, TRX 1, Timeslot 0, Lchan 0: Type NONE
  Connection: 0, State: BROKEN UNUSABLE Error reason: activation timeout
  BS Power: 35 dBm, MS Power: 33 dBm
  Channel Mode / Codec: signalling
  No Subscriber
  Bound IP: 0.0.0.0 Port 0 RTP_TYPE2=0 CONN_ID=0
  Measurement Report:
    Flags: DLinval 
    RXL-FULL-ul: -110 dBm, RXL-SUB-ul: -110 dBm RXQ-FULL-ul: 0, RXQ-SUB-ul: 0

Interestingly, so far I've only seen TCH channels stuck and I've never seen SDCCH channels stuck.

I tried to analyze whether something else was happening at the time of the issue and it looks like another BTS had a spike in traffic making both of its 2 TRX saturated AND we had relatively high load average on the server at the same time (1.2 vs normal 0.2-0.5) which makes me think it's IO related. I think what may be happening is that a (relatively) high time waiting for our SQLite DB may have led to a timeout and stuck the channel. Not sure how realistic this is.

Happy to run some more debugging or provide NITB/BTS logs if there is any interest in fixing this.

PS I also see an OML/RSL fail at around the same time, so I wonder if this is related. I failed to find this moment in the logs after a quick look, though, so can't correlate. =|

@ipse/all: Please see: http://lists.osmocom.org/pipermail/openbsc/2017-April/010560.html.

One issue is if the SACCH deactivation ack arrives after the timeout (e.g. because it is delayed through TCP) we will try to release the channel again and as it fails mark the channel as broken. It should only be a couple of hours of work (it can be easily simulated with my Smalltalk fakebts code or newer approaches, ttcn-3?) but sadly the combination of day job + commute keep me busy enough.

Alex, I hope that fairwaves finds the time to move the lchan to use a FSM or can sponsor that work? I am afraid that my non OpenBSC/Osmocom paid work and commute don't leave too much time to fix such issues.

Rhizomatica installations have had

    timer t3109 4

After patches from zecke a few months back, we now recover all stuck or "broken" channels except for those resulting from NACK on activation.
I currently use a cronjob to wait for a quiet moment and then drop the bts when these accumulate.
I found it hard to impossible to reproduce anything in the production environment, or to track down any identifiable cause, so I kind of gave up on it for the moment.