https://projects.osmocom.org/https://projects.osmocom.org/favicon.ico?16647414092022-10-01T11:20:40ZOpen Source Mobile CommunicationsOCTOI - Osmocom Community TDM over IP - Bug #5694: BERT testinghttps://projects.osmocom.org/issues/5694?journal_id=249822022-10-01T11:20:40Zlaforge
<ul></ul><p>Some testing of PrTel93i via Auerswald PBX to DAHDI/freeswitch (retronetworking Event setup) and back has shown 15min with zero BER.</p> OCTOI - Osmocom Community TDM over IP - Bug #5694: BERT testinghttps://projects.osmocom.org/issues/5694?journal_id=249862022-10-02T21:18:20Zlaforge
<ul><li><strong>% Done</strong> changed from <i>0</i> to <i>20</i></li></ul><p>laforge wrote in <a href="#note-1">#note-1</a>:</p>
<blockquote>
<p>Some testing of PrTel93i via Auerswald PBX to DAHDI/freeswitch (retronetworking Event setup) and back has shown 15min with zero BER.</p>
</blockquote>
<p>This was confirmed several times yesterday and today. IT still worked after upgrading DAHDI from dahdi 3.1.0 to the dahdi-linux laforge/trunkdev. The only time I saw a bit error was when enabling/disabling RXMIRROR/TXMIRROR <strong>after</strong> the call had started. Starting captures before (and terminating after) the PrTel call has 0 BER.</p>
<p>traces of the BER pattern are at <a class="external" href="https://people.osmocom.org/laforge/retronetworking/prbs/">https://people.osmocom.org/laforge/retronetworking/prbs/</a> - we can see that there's a 0x5D pattern for synchronization before the PRBS sequence starts.</p>
<p>According to <a class="user active" href="https://projects.osmocom.org/users/293278">manawyrm</a> the PRBS sequence matches the prbs11 we have in osmo-prbs/libosmocore.</p>
Some ideas
<ul>
<li>implement a raw loop type channel in yate so we can do Prtel->yate and back to avoid testing the path twice</li>
<li>implement a BER test channel in yate so we can do the same BER testing without a PrTel</li>
</ul>
<p>I've started to read up, hopefully understand and hack my first yate module implementing source/consumer/channel/driver classes. Put it aside for now as other bits were more important.</p> OCTOI - Osmocom Community TDM over IP - Bug #5694: BERT testinghttps://projects.osmocom.org/issues/5694?journal_id=250942022-10-12T21:11:34Zmanawyrm
<ul></ul><p>I offer a PRBS11 BERT at 030 6502 8003. <br />It's just a 128M .alaw file with pre-generated PRBS sequences. <br />The file content was validated using an ARGUS 4 tester (which showed no bit errors over the first 15 min of the call).</p>
<p>Playback is done using Yate's internal wave player.</p>
<p>Then I've done some additional testing, such as removing the icE1usb completely from the signal path.<br />My setup looks a lot like the Nuremburg hub now. Uplink to OCTOI is now being done using trunkdev.<br />Clocking is still provided via a port on the TE820 from the icE1usb, but no data is passing through the icE1usb.</p>
<p>validated, 0 BER setups:<br />ARGUS 4 -> Auerswald -> TE820 -> Yate PBX -> TE820 -> Auerswald -> ARGUS 4 (different B-channel)<br />ARGUS 4 -> Auerswald -> TE820 -> Yate PRBS11 (also when using both B-channels)</p>
<p>non working, high BER setups:<br />ARGUS 4 -> Auerswald -> TE820 -> Yate PBX -> icE1usb -> OCTOI DIVF -> icE1usb -> Yate PRBS11<br />ARGUS 4 -> Auerswald -> TE820 -> Yate PBX -> trunkdev -> OCTOI DIVF -> trunkdev -> Yate PRBS11</p>
<p>untested setups:<br />ARGUS 4 -> Auerswald -> TE820 -> Yate PBX -> icE1usb -> OCTOI DIVF -> icE1usb -> Auerswald -> ARGUS 4 (something about the signalling gets messed up along the way, ARGUS tester doesn't recognize that the incoming call is itself and just rings like a phone)</p>
<p>what have we learned:<br />icE1usb seems to be innocent, a pure trunkdev setup also exhibits the bit errors.<br />I believe this testing restricts the possible problem points to just OCTOI/osmo-e1d itself.<br />This is a rather brave assumption, but we've already seen this behaviour before the move to Nuremburg, so I don't think the new HW or the added VM layer are problematic.<br />My 0 BER setup also runs inside a VM, so that should hopefully be fine.</p>
<p>I have added some rudimentary debugging (just dumping the raw data into a file) on both the input and output of the RIFO code. <br />No errors and differences were noticable (no reordering occured while this test was running).</p>
<p>My ISDN tester also beeps whenever an error is detected, which is useful for correlating bit errors to osmo-e1d logs. <br />It doesn't beep when reordering occurs. It does beep when packet loss occurs. This is exactly as expected and a good sign :)</p>
<p>I've also timed the delay between errors (which always seem to come in bundles) and couldn't find any noticable systematic timing. Seems to be somewhat random.</p>
<p><a class="user active" href="https://projects.osmocom.org/users/7">laforge</a> As you're the author of the e1d/OCTOI code: Can you think of any other places in the e1d code where something like this could occur (which would affect both icE1 and trunkdev)?</p> OCTOI - Osmocom Community TDM over IP - Bug #5694: BERT testinghttps://projects.osmocom.org/issues/5694?journal_id=250952022-10-13T06:01:49Zlaforge
<ul></ul><p>Thanks for your testing.</p>
<p>Please note that we (tnt and I) did quite a bit of testing with PRBS testing on all timeslots, without yate/ISDN on top. The code for that is in <code>osmo-e1d/e1-prbs-test</code>. As far as I remember there were no PRBS issues discovered in either real E1 (between 2x icE1usb, or between TE802 and icE1usb) nor over OCTOI. At least not at the time. I guess the best way to start is from the bottom up by re-doing those tests in a variety of configurations. If those tests still work while your ISDN-based single-TS tests fail, it must be something about the yate/dahdi integration.</p> OCTOI - Osmocom Community TDM over IP - Bug #5694: BERT testinghttps://projects.osmocom.org/issues/5694?journal_id=250962022-10-13T06:04:18Zlaforge
<ul></ul><p>Maybe it would be worth to</p>
<ul>
<li>add stats exporter code to e1-prbs-check (so we get prbs related info into grafana)</li>
<li>set up a "prbs" user on divf</li>
<li>continuously run e1-prbs-test on the associated trunkdev</li>
</ul>
<p>This way anyone could connect as prbs user and could get PRBS results for all TS in the same timeline as packet re-order/loss statistics.</p> OCTOI - Osmocom Community TDM over IP - Bug #5694: BERT testinghttps://projects.osmocom.org/issues/5694?journal_id=251912022-10-22T20:00:50Zlaforge
<ul></ul><p><a class="user active" href="https://projects.osmocom.org/users/293278">manawyrm</a> what your testing didn't rule out is some other problem with DIVF specifically, such as the interrupt misses likely caused by virtualization. All your tests involving osmo-e1d/octoi-protocol also involved the DIVF instance.</p> OCTOI - Osmocom Community TDM over IP - Bug #5694: BERT testinghttps://projects.osmocom.org/issues/5694?journal_id=251922022-10-22T21:40:15Zmanawyrm
<ul></ul><p>laforge wrote in <a href="#note-6">#note-6</a>:</p>
<blockquote>
<p><a class="user active" href="https://projects.osmocom.org/users/293278">manawyrm</a> what your testing didn't rule out is some other problem with DIVF specifically, such as the interrupt misses likely caused by virtualization. All your tests involving osmo-e1d/octoi-protocol also involved the DIVF instance.</p>
</blockquote>
<p>Indeed. I'm aware and I want to build a simpler (local) setup next and try to build a DIVF clone and see if the trouble still occurs. <br />I do have a similar AMD Epyc machine available, I do have the TE820 (well, only 1), but I think I might be able to get something to happen.</p>
<p>Currently away from my setup for another week again...</p> OCTOI - Osmocom Community TDM over IP - Bug #5694: BERT testinghttps://projects.osmocom.org/issues/5694?journal_id=251942022-10-23T16:02:17Zlaforge
<ul><li><strong>Status</strong> changed from <i>New</i> to <i>In Progress</i></li></ul><p>I'd really like to have some kind of device or program or setup containing of any combination of software and hardware where we can do automatic (scripted) ISDN B-channel BERT. Doing this manually on a PrTel or Argus just doesn't scale.</p>
<p>some months ago, I was trying to write some "simple" testing code using libcapi20 (see <a class="external" href="https://gitea.osmocom.org/retronetworking/capi-hacks">https://gitea.osmocom.org/retronetworking/capi-hacks</a>). It contains a simple "B channel loop" mode where all data received on a b-channel is echoed back to it. I was trying to use this with BERT testing, but unfortuantely it created significant (E-2/E-3) BER rates via a hfc-usb / mISDN / capi20 path.</p>
<p>Today I tried the same using the rcapi protocol of a <a class="wiki-page" href="https://projects.osmocom.org/projects/retronetworking/wiki/Bintec_X1200">Bintec_X1200</a>. Unfortuantely with the sam results: Very high (E-2/E-3) BER is reported. I initially thought this might be due to the lack of TCP_NODELAY in the rcapi module of libcapi20, but even when adding that setsockopt call, I still had the high BER.</p>
<p>When making a voice call and speaking, the BER is not audible.</p> OCTOI - Osmocom Community TDM over IP - Bug #5694: BERT testinghttps://projects.osmocom.org/issues/5694?journal_id=252352022-10-26T12:42:42Zmanawyrm
<ul><li><strong>File</strong> <a href="/attachments/5498">hbsldfqjixg.jpg</a> <a class="icon-only icon-download" title="Download" href="/attachments/download/5498/hbsldfqjixg.jpg">hbsldfqjixg.jpg</a> added</li><li><strong>File</strong> <a href="/attachments/5499">qlcdrxgnmvj.jpg</a> <a class="icon-only icon-download" title="Download" href="/attachments/download/5499/qlcdrxgnmvj.jpg">qlcdrxgnmvj.jpg</a> added</li></ul><p>OK, I did some more BERT testing.</p>
<p>Little bit different setup, this time with the HFC-S PCI OCTOI setup (<a class="external" href="https://osmocom.org/projects/octoi/wiki/Trunkdev-S0-Adapter">https://osmocom.org/projects/octoi/wiki/Trunkdev-S0-Adapter</a> ).<br />HFC-S -> Asterisk -> osmo-e1d (trunkdev) -> Internet (Alfeld, Telekom -> Kiel, Versatel) -> osmo-e1d (trunkdev) -> Yate</p>
<p>Yate is replying with the .alaw PRBS11 file again. Connection was absolutely spotless (see attached 15:00min BERT photos).</p>
<p>My yate is running on a Debian 11 VM, clocked by an icE1usb on Port 1 of a TE820 (which is passed through to the VM).<br />Host is Debian 11 as well, Intel i5-10400 CPU, qemu-kvm & libvirt for virtualization. <br />Default kernel parameters on the host.<br />The VM is using "pcie_aspm=off pcie_port_pm=off" (which was leftover from all the PCIe->PCI bridge testing)</p>
<p>As far as I can tell, the main differences are: <br />- Intel instead of AMD CPU<br />- 1x TE820 card instead of 2<br />- PCIe PM disabled on the guest</p>
<p>(I'm also not seeing any of those lost interrupt errors. I've correlated the timing of the messages in dmesg on DIVF with the errors in the B-channel and they do <em>not</em> match!)</p>
<p>My host machine is using 5.16.0-0.bpo.4-amd64 (from backports, due to the very recent hardware), the guest VM is using 5.10.0-11-amd64.</p>
<p>Some ideas:<br />- disable the second TE820<br />- update the host kernel to Debian backports<br />- disable PCIe ASPM (maybe on both host and guest)</p>
<p>I have access to some AMD machines of the same generation, unfortunately without TE820. Maybe worth a try to replicate the problem there?</p> OCTOI - Osmocom Community TDM over IP - Bug #5694: BERT testinghttps://projects.osmocom.org/issues/5694?journal_id=252362022-10-26T15:33:00Zmanawyrm
<ul></ul><p>Just switched the Kiel and Alfeld setups back to the Berlin node at LaF0rge.<br />Interestingly enough: Same problem.</p>
<p>HFC-S -> Asterisk -> osmo-e1d (trunkdev) -> Internet (Alfeld, Telekom -> Berlin, Vodafone) -> osmo-e1d (trunkdev) -> Yate -> osmo-e1d (physical icE1usb) -> osmo-e1d (trunkdev) -> Yate<br />shows high bitrate again. So the problem even occurs on the Berlin machine. Similar CPU, similar Yate config, only 1 TE820.</p>
<p>Interesting...</p> OCTOI - Osmocom Community TDM over IP - Bug #5694: BERT testinghttps://projects.osmocom.org/issues/5694?journal_id=258832022-12-31T20:24:15Zlaforge
<ul><li><strong>Status</strong> changed from <i>In Progress</i> to <i>Stalled</i></li></ul>