https://projects.osmocom.org/https://projects.osmocom.org/favicon.ico?16647414092019-08-19T08:00:38ZOpen Source Mobile CommunicationsOsmoBSC - Bug #4156: osmo-bsc test suite regression since build 694https://projects.osmocom.org/issues/4156?journal_id=156242019-08-19T08:00:38Zosmith
<ul><li><strong>Status</strong> changed from <i>New</i> to <i>In Progress</i></li></ul> OsmoBSC - Bug #4156: osmo-bsc test suite regression since build 694https://projects.osmocom.org/issues/4156?journal_id=156262019-08-19T11:48:55Zosmith
<ul></ul><p>There are three types of errors:</p>
a) 5 builds ago (2019-08-14)
<ul>
<li>affected:
<ul>
<li>ttcn3-bsc-test-latest</li>
<li>ttcn3-bsc-test</li>
</ul>
</li>
<li>failing tests:
<ul>
<li>TC_ctrl</li>
<li>TC_oml_unknown_unit_id</li>
</ul></li>
</ul>
b) 5 builds ago (2019-08-14)
<ul>
<li>affected:
<ul>
<li>ttcn3-bsc-test-sccplite</li>
</ul>
</li>
<li>failing tests:
<ul>
<li>TC_ho_int</li>
<li>TC_ho_out_fail_*</li>
<li>TC_ho_out_of_this_bsc</li>
</ul></li>
</ul>
c) 1 build ago (2019-08-19)
<ul>
<li>affected:
<ul>
<li>ttcn3-bsc-test-sccplite</li>
<li>ttcn3-bsc-test-sccplite-latest</li>
<li>ttcn3-bsc-test-latest</li>
<li>ttcn3-bsc-test</li>
</ul>
</li>
<li>failing tests:
<ul>
<li>TC_assignment_csd</li>
<li>TC_assignment_ctm</li>
<li>all from TC_bssap_rsld_does_not_cause_bssmap_reset to TC_chan_rel_rll_rel_ind</li>
<li>... (many more!)</li>
</ul></li>
</ul>
<p>Regarding c): I've reproduced it locally, and found that reverting commit "RAN_Adapter: invert check for RAN_ops == omit" in osmo-ttcn3-hacks.git fixes it: <a class="external" href="https://gerrit.osmocom.org/c/osmo-ttcn3-hacks/+/15233">https://gerrit.osmocom.org/c/osmo-ttcn3-hacks/+/15233</a></p>
<p>Regarding b): I'm assuming that we don't care if handover related tests pass with sccplite, so this can be ignored. It seems that the tests were passing before, although they should not have been passing, and now they are correctly failing.</p>
<p>Regarding a): "5 builds ago" is when new TC_ho_neighbor_config_* tests were introduced, and related osmo-bsc config changes were made (see <a class="external" href="https://gerrit.osmocom.org/q/topic:ho">https://gerrit.osmocom.org/q/topic:ho</a> ). Note that the sccplite related tests are not failing, so this might be a config related issue (the sccplite tests have their own osmo-bsc.cfg, and that one was not modified).</p>
<p>So only a) is remaining, I'm on it :)</p> OsmoBSC - Bug #4156: osmo-bsc test suite regression since build 694https://projects.osmocom.org/issues/4156?journal_id=156272019-08-19T12:45:30Zosmith
<ul></ul>Patches to fix TC_ctrl from a):
<ul>
<li><a class="external" href="https://gerrit.osmocom.org/c/docker-playground/+/15235">https://gerrit.osmocom.org/c/docker-playground/+/15235</a></li>
<li><a class="external" href="https://gerrit.osmocom.org/c/osmo-ttcn3-hacks/+/15234">https://gerrit.osmocom.org/c/osmo-ttcn3-hacks/+/15234</a></li>
</ul> OsmoBSC - Bug #4156: osmo-bsc test suite regression since build 694https://projects.osmocom.org/issues/4156?journal_id=156362019-08-19T14:09:42Zosmith
<ul><li><strong>% Done</strong> changed from <i>0</i> to <i>80</i></li></ul><p>The TC_oml_unknown_unit_id failure is caused by changes in docker-playground.git, it passed locally when I checked out a ~3 weeks old commit ef8b48516e0d5215a92dfe6de32983a450f4ed42.</p> OsmoBSC - Bug #4156: osmo-bsc test suite regression since build 694https://projects.osmocom.org/issues/4156?journal_id=156462019-08-20T13:31:31Zosmith
<ul></ul><p>Fix for TC_oml_unknown_unit_id: <a class="external" href="https://gerrit.osmocom.org/c/osmo-ttcn3-hacks/+/15254">https://gerrit.osmocom.org/c/osmo-ttcn3-hacks/+/15254</a></p> OsmoBSC - Bug #4156: osmo-bsc test suite regression since build 694https://projects.osmocom.org/issues/4156?journal_id=156472019-08-20T13:38:22Zosmith
<ul></ul><p>Now that the > 60 failing tests we had yesterday are resolved, it appears that we have one more failing test since yesterday: TC_paging_resp_unsol</p>
<p>This should be the last one, after which everything is back to like it was before this regression wave.</p>
<p>I'm looking into resolving this one too.</p> OsmoBSC - Bug #4156: osmo-bsc test suite regression since build 694https://projects.osmocom.org/issues/4156?journal_id=156552019-08-21T06:27:48Zosmith
<ul><li><strong>Status</strong> changed from <i>In Progress</i> to <i>Resolved</i></li><li><strong>% Done</strong> changed from <i>80</i> to <i>100</i></li></ul>With git bisect I found that the TC_paging_resp_unsol failure is expected: the test-expectation has been changed by <a class="user active" href="https://projects.osmocom.org/users/15">dexter</a>, but the patch that adjusts osmo-bsc behavior was not merged yet:
<ul>
<li><a class="external" href="https://gerrit.osmocom.org/c/osmo-ttcn3-hacks/+/15208">https://gerrit.osmocom.org/c/osmo-ttcn3-hacks/+/15208</a></li>
<li><a class="external" href="https://gerrit.osmocom.org/c/osmo-bsc/+/15209">https://gerrit.osmocom.org/c/osmo-bsc/+/15209</a></li>
</ul>
<p>Maybe it would have been better to wait with merging the osmo-ttcn3-hacks patch until the osmo-bsc patch was merged?</p>
<p><a class="user active" href="https://projects.osmocom.org/users/7">laforge</a> wrote:</p>
<blockquote>
<p>we also need to figure out how related patches ended up untested in the code tree in the first place.</p>
</blockquote>
<p>The patch that caused the most fall out is this library change:<br /><a class="external" href="https://gerrit.osmocom.org/c/osmo-ttcn3-hacks/+/15157">https://gerrit.osmocom.org/c/osmo-ttcn3-hacks/+/15157</a></p>
<p>It looks good, but since we don't automatically run the TTCN3 tests in gerrit verification, it was not clear that this breaks the BSC testsuite. From reading the commit message, it seems like it was probably manually tested against SGSN tests, and testing manually against all testsuites does not seem feasible.</p>
<p>I understand that it is not feasible right now to run the TTCN3 tests in the gerrit verification job either (due to the long execution times, especially if there is a library change), but hopefully we can get there eventually? I would be happy to work on this if it seems realistic and we can fit it in at some point in the future.</p>
<p>All related patches to fix the regressions are merged, setting this issue to resolved.<br /><a class="external" href="https://gerrit.osmocom.org/q/topic:ttcn3-fix-bsc">https://gerrit.osmocom.org/q/topic:ttcn3-fix-bsc</a></p>