Bug #4903
closedTons of TTCN3 test failures after osmo-gbproxy "rewrite" patch merge
100%
Description
See https://jenkins.osmocom.org/jenkins/view/TTCN3/job/ttcn3-gbproxy-test-fr/ and https://jenkins.osmocom.org/jenkins/view/TTCN3/job/ttcn3-gbproxy-test/
I've been trying to debug this for more than one day now, mostly being delayed by:- existing tcpdump not being able to capture HDLC traffic properly
- pcap file format insufficient for captures of interfaces with multiple link types
- switching to dumpcap revealed tons of problems
- dumpcap not present in images
- dumpcap unable to write to /data in the containers for no known reason
This is all finally resolved now and we get proper .pcapng.gz files as artefacts of the jenkins jobs.
- The test fail with "Timeout waiting for unblock of all BVCs", new component reason: "Timeout waiting for unblock of all BVCs"
- a review of the pcap file confirms this: We see SIG + PTP resets on the RAN side, but only SIG BVC RESET on the SGSN side
- I cannot reproduce those failures when executing the tests locally on my laptop, even with HDLC/dahdi
- When executing single test cases using the exact same (not rebuilt) docker containres on the very same build slave, the tests pass
- when executing the same test case 7 times in a row by those exact same containerse, they also all pass
So all in all, this hints to something related to left-over state in gbproxy between test cases leading to failure by following test cases.
This is plausible as there are now additional objects like the gbproxy_cell and the per-NSE gbproxy_bvc which we may want to clean-up between test runs.
Files
Related issues
Updated by laforge over 3 years ago
So the culprit seems to be GBProxy_Tests.TC_bvc_reset_sig_from_sgsn
Tests before this one have a chance of passing, afterwards they appear to all fail.
Updated by laforge over 3 years ago
- % Done changed from 20 to 30
after a "normal" test case termination:
OsmoGbProxy> show gbproxy bss NSEI 2003, PTP-BVCI 0 [UNBLOCKED] NSEI 2003, PTP-BVCI 20031, RAI 262-42-13135-1 [UNBLOCKED] NSEI 2003, PTP-BVCI 20032, RAI 262-42-13300-0 [UNBLOCKED] NSEI 2003, PTP-BVCI 20033, RAI 262-42-13300-0 [UNBLOCKED] NSEI 2001, PTP-BVCI 0 [UNBLOCKED] NSEI 2001, PTP-BVCI 20011, RAI 262-42-13135-0 [UNBLOCKED] NSEI 2002, PTP-BVCI 0 [UNBLOCKED] NSEI 2002, PTP-BVCI 20021, RAI 262-42-13135-1 [UNBLOCKED] NSEI 2002, PTP-BVCI 20022, RAI 262-42-13135-2 [UNBLOCKED] OsmoGbProxy> show gbproxy sgsn NSEI 101, PTP-BVCI 0 [UNBLOCKED] NSEI 101, PTP-BVCI 20031, RAI 000-000-0-0 [UNBLOCKED] NSEI 101, PTP-BVCI 20021, RAI 000-000-0-0 [UNBLOCKED] NSEI 101, PTP-BVCI 20011, RAI 000-000-0-0 [UNBLOCKED] NSEI 101, PTP-BVCI 20032, RAI 000-000-0-0 [UNBLOCKED] NSEI 101, PTP-BVCI 20022, RAI 000-000-0-0 [UNBLOCKED] NSEI 101, PTP-BVCI 20033, RAI 000-000-0-0 [UNBLOCKED]
however, after termination of TC_bvc_reset_sig_from_sgsn:
OsmoGbProxy> show gbproxy bss NSEI 2003, PTP-BVCI 0 [UNBLOCKED] NSEI 2003, PTP-BVCI 20031, RAI 262-42-13135-1 [UNBLOCKED] NSEI 2003, PTP-BVCI 20032, RAI 262-42-13300-0 [UNBLOCKED] NSEI 2003, PTP-BVCI 20033, RAI 262-42-13300-0 [UNBLOCKED] NSEI 2001, PTP-BVCI 0 [UNBLOCKED] NSEI 2001, PTP-BVCI 20011, RAI 262-42-13135-0 [UNBLOCKED] NSEI 2002, PTP-BVCI 0 [UNBLOCKED] NSEI 2002, PTP-BVCI 20021, RAI 262-42-13135-1 [UNBLOCKED] NSEI 2002, PTP-BVCI 20022, RAI 262-42-13135-2 [UNBLOCKED] OsmoGbProxy> show gbproxy sgsn NSEI 101, PTP-BVCI 0 [UNBLOCKED]
So the SGSN-side PTP BVCs have been removed as expected. However, when starting any other test-case afterwards, they are not being re-created as the PTP BVC resets are coming in from the BSS side.
Updated by laforge over 3 years ago
- % Done changed from 30 to 70
Proposed fix in https://gerrit.osmocom.org/c/osmo-sgsn/+/21689
Updated by laforge over 3 years ago
- Related to Bug #4904: osmo-gbproxy segfault in GBProxy_Tests.TC_paging_ps_sig_lac_unknown added
Updated by laforge over 3 years ago
- Status changed from In Progress to Resolved
- % Done changed from 70 to 100
