Bug #2335
closedSteady increasing clock drift in RTP from source BTS
100%
Description
While analyzing some of the packet captures, Pau encountered some serious issues with the clock of the source of the RTP stream drifting quickly towards high values such as 5000msecs of difference.
Wireshark was used on relatded protocol traces.
Select that one on the "RTP Streams" window and Press the button "Analyze". In there you can see a "Clock Drift: -5507 ms", and "Max Skew: -4040.34 ms" for a session which lasts 314 seconds.
In the list of packets on the same window, you can see the column "Skew" steady increasing over time (with minor instantaneous increases/decreases but with a steady increasing tendency). It seems if the call would have been longer, this drift would continue growing forever.
I also saw similar behaviour while running the pcap traces through the de-jitter implementation. At some point the de-jitter isreceiving packets arriving too late in time, and it starts dropping them. Then it increases the buffer size as it detects it as jitter, and it can continue a bit more until the issue happens again with the bigger buffer. It continues regulating itself until it reaches the maximum permitted buffer size configured. After that, it starts dropping all packets because all packets arrive too late. Looking at the graphs of how packets were delayed by the dejitter I had the impression the clock drift was appearing, so I added support to calculate an estimated drift from the source andit can take the drifting into account and can plot that too now. I attach a screenshot of the plot here for reference.
In the plot, you can see the red descending line which is the estimated skew from the source in milliseconds over time. It is calculated by smoothing the instantaneous delay of each packet towards the initial packet. In this plot as the de-jitter already takes into account the skew, packets are usually not dropped anymore. They are only dropped during the huge change of the skew in the middle of the plot because the smoothed value cannot represent correctly those enormous instantaneous checks, so it detects that as variability and the de-jitter increases the buffer size to account for that.
So even if the de-jitter can handle this better than before, it is still a main issue because the delay of several seconds is still there.
Files
Checklist
- determine osmo-bts version showing this
- check if current mastere also shows this
- ensure we always get PH-DATA.ind on each PHY for each codec frame, even if only junk was received (DTX, inteference, idle channel, ...)
- always return GSM_RTP_DURATION when creating RTP packet
- transform fn_ms_adj() to a pure checking function that raises alarms / prints logs whenever we're seeing missing codecx frames in l1sap_tch_ind()
Related issues
