Bug #4708
closedVarious SMSCB test cases fail with frequency hopping enabled
100%
Description
See https://jenkins.osmocom.org/jenkins/view/TTCN3/job/ttcn3-bts-test/test_results_analyzer/ (but also in latest).
We have new failures in:- TC_cbc_sdcch4_load_overload
- TC_cbc_sdcch8_load_overload
- TC_etws_p1ro
- TC_etws_p1ro_end
- TC_sms_cb_cmd_sdcch4_4block
- TC_sms_cb_cmd_sdcch4_default_and_normal
- TC_sms_cb_cmd_sdcch4_multi
- TC_sms_cb_cmd_sdcch8_1block
- TC_sms_cb_cmd_sdcch8_4block
- TC_sms_cb_cmd_sdcch8_default_and_normal
- TC_sms_cb_cmd_sdcch8_multi
Related issues
Updated by laforge over 3 years ago
A number of them are "VTY timeout for prompt" which doesn't sount good.
Updated by laforge over 3 years ago
- Priority changed from High to Urgent
Looking at the junit xml at https://jenkins.osmocom.org/jenkins/view/TTCN3/job/ttcn3-bts-test/983/console it seems TC_sabm_incorrect_c is the last correctly executed test case, and afterwards all further tests (starting with BTS_Tests:hopping.TC_chan_act_stress) all tests fail with either "Timeout waiting for RSL bring up" or "VTY Timeout for prompt"
Updated by fixeria over 3 years ago
Interestingly enough, I could not reproduce those failures on my machine in Docker:
pass BTS_Tests_SMSCB.TC_cbch_load_idle_no_cbch pass BTS_Tests_SMSCB.TC_sms_cb_cmd_sdcch4_1block pass BTS_Tests_SMSCB.TC_sms_cb_cmd_sdcch4_2block pass BTS_Tests_SMSCB.TC_sms_cb_cmd_sdcch4_3block pass BTS_Tests_SMSCB.TC_sms_cb_cmd_sdcch4_4block pass BTS_Tests_SMSCB.TC_sms_cb_cmd_sdcch4_multi pass BTS_Tests_SMSCB.TC_sms_cb_cmd_sdcch4_schedule pass BTS_Tests_SMSCB.TC_sms_cb_cmd_sdcch4_default_only pass BTS_Tests_SMSCB.TC_sms_cb_cmd_sdcch4_default_and_normal pass BTS_Tests_SMSCB.TC_sms_cb_cmd_sdcch4_default_then_null pass BTS_Tests_SMSCB.TC_cbc_sdcch4_load_idle pass BTS_Tests_SMSCB.TC_cbc_sdcch4_load_overload pass BTS_Tests_SMSCB.TC_etws_p1ro pass BTS_Tests_SMSCB.TC_etws_p1ro_end pass BTS_Tests_SMSCB.TC_etws_pcu NEW: PASS BTS_Tests_SMSCB.TC_sms_cb_cmd_sdcch8_1block NEW: PASS BTS_Tests_SMSCB.TC_sms_cb_cmd_sdcch8_2block NEW: PASS BTS_Tests_SMSCB.TC_sms_cb_cmd_sdcch8_3block NEW: PASS BTS_Tests_SMSCB.TC_sms_cb_cmd_sdcch8_4block NEW: PASS BTS_Tests_SMSCB.TC_sms_cb_cmd_sdcch8_multi NEW: PASS BTS_Tests_SMSCB.TC_sms_cb_cmd_sdcch8_schedule NEW: PASS BTS_Tests_SMSCB.TC_sms_cb_cmd_sdcch8_default_only NEW: PASS BTS_Tests_SMSCB.TC_sms_cb_cmd_sdcch8_default_and_normal NEW: PASS BTS_Tests_SMSCB.TC_cbc_sdcch8_load_idle NEW: PASS BTS_Tests_SMSCB.TC_cbc_sdcch8_load_overload
Updated by fixeria over 3 years ago
I also tried to run BTS_Tests_SMSCB.control exclusively in Jenkins [1], all SMSCB test cases passed.
The reason why they fail when ttcn3-bts-test is started by timer remains a mystery.
[1] https://jenkins.osmocom.org/jenkins/view/TTCN3-centos/job/TTCN3-centos-bts-test/88/
Updated by fixeria over 3 years ago
- Category set to osmo-bts-trx
- Status changed from In Progress to Feedback
- Priority changed from Urgent to Normal
Apparently, this is not a regression. I think the problem was that before [1] jenkins.sh did not append ':hopping' to both BTS_Tests_{SMSCB,LAPDm} groups (sorry, my fault), so the hopping test results got mixed somehow with non-hopping ones in junit-*.xml. Since [1] has been merged, non-hopping SMSCB test cases passing two days on Jenkins as before [2][3][4].
[1] https://gerrit.osmocom.org/c/docker-playground/+/19637
[2] https://jenkins.osmocom.org/jenkins/view/TTCN3/job/ttcn3-bts-test/991/
[3] https://jenkins.osmocom.org/jenkins/view/TTCN3/job/ttcn3-bts-test-latest/677/
[4] https://jenkins.osmocom.org/jenkins/view/TTCN3-centos/job/TTCN3-centos-bts-test/89/
Updated by fixeria over 3 years ago
- Subject changed from Various SMSCB tests fail since last build (August 13) to Various SMSCB test cases fail with frequency hopping enabled
- Status changed from Feedback to New
- Priority changed from Normal to Low
We should still investigate why the hopping test cases fail and fix them. Hijacking this ticket, setting priority to 'Low' because I am pretty sure that it's not a problem in the IUT, but rather in the test suite itself (e.g. BTS_Tests_SMSCB.f_smscb_setup() does not consider hopping parameters).
Updated by fixeria over 3 years ago
- Related to Feature #4546: baseband frequency hopping support for osmo-bts-trx added
Updated by laforge over 3 years ago
- Priority changed from Low to Normal
setting priority to normal. It doesn't matter whether it's a bug in osmo-bts or in the test suite. We cannot prove it ither way unless we fix it. As neels is currently working on a sysmocom-custmer ticket that aims to validate SMSCB is fully functional, I'm sure at least he (and the unnamed sysmocom customer) would greatly appreciate if the tests passed. Particularly as they are using frequency hopping.
Updated by neels over 3 years ago
i'm so far only working on SMSCB in OsmoBSC, if I should continue in OsmoBTS then let me know ... !
Updated by fixeria over 3 years ago
i'm so far only working on SMSCB in OsmoBSC, if I should continue in OsmoBTS then let me know ... !
Since I am more familiar with the related code in ttcn3-bts-test, I'll take care of it. Thanks anyway!
Updated by fixeria over 3 years ago
- Status changed from In Progress to Feedback
- % Done changed from 0 to 100
Please see:
https://gerrit.osmocom.org/c/docker-playground/+/20098 ttcn3-bts-test/fh: do not configure CBCH slots statically
and
https://gerrit.osmocom.org/c/osmo-ttcn3-hacks/+/20100 BTS_Tests: simplify MA length calculation in f_resolve_fh_params()
https://gerrit.osmocom.org/c/osmo-ttcn3-hacks/+/20101 BTS_Tests: group hopping parameters into a separate record
https://gerrit.osmocom.org/c/osmo-ttcn3-hacks/+/20102 BTS_Tests_SMSCB: handle frequency hopping parameters
Here are the results from a test run on my machine:
pass BTS_Tests_SMSCB.TC_cbch_load_idle_no_cbch pass BTS_Tests_SMSCB.TC_sms_cb_cmd_sdcch4_1block pass BTS_Tests_SMSCB.TC_sms_cb_cmd_sdcch4_2block pass BTS_Tests_SMSCB.TC_sms_cb_cmd_sdcch4_3block pass BTS_Tests_SMSCB.TC_sms_cb_cmd_sdcch4_4block pass BTS_Tests_SMSCB.TC_sms_cb_cmd_sdcch4_multi pass BTS_Tests_SMSCB.TC_sms_cb_cmd_sdcch4_schedule pass BTS_Tests_SMSCB.TC_sms_cb_cmd_sdcch4_default_only pass BTS_Tests_SMSCB.TC_sms_cb_cmd_sdcch4_default_and_normal pass BTS_Tests_SMSCB.TC_sms_cb_cmd_sdcch4_default_then_null pass BTS_Tests_SMSCB.TC_cbc_sdcch4_load_idle pass BTS_Tests_SMSCB.TC_cbc_sdcch4_load_overload pass BTS_Tests_SMSCB.TC_etws_p1ro pass BTS_Tests_SMSCB.TC_etws_p1ro_end pass BTS_Tests_SMSCB.TC_etws_pcu NEW: PASS BTS_Tests_SMSCB.TC_sms_cb_cmd_sdcch8_1block NEW: PASS BTS_Tests_SMSCB.TC_sms_cb_cmd_sdcch8_2block NEW: PASS BTS_Tests_SMSCB.TC_sms_cb_cmd_sdcch8_3block NEW: PASS BTS_Tests_SMSCB.TC_sms_cb_cmd_sdcch8_4block NEW: PASS BTS_Tests_SMSCB.TC_sms_cb_cmd_sdcch8_multi NEW: PASS BTS_Tests_SMSCB.TC_sms_cb_cmd_sdcch8_schedule NEW: PASS BTS_Tests_SMSCB.TC_sms_cb_cmd_sdcch8_default_only NEW: PASS BTS_Tests_SMSCB.TC_sms_cb_cmd_sdcch8_default_and_normal NEW: PASS BTS_Tests_SMSCB.TC_cbc_sdcch8_load_idle NEW: PASS BTS_Tests_SMSCB.TC_cbc_sdcch8_load_overload
Updated by fixeria over 3 years ago
- Status changed from Feedback to Resolved
All changes have been merged.