Cellular Network Infrastructure

still happening https://jenkins.osmocom.org/jenkins/job/ttcn3-ggsn-test/262/console

I cannot reproduce this problem locally. I have tried to reprocuce it by running the first two GGSN tests in a loop.

I suspect this issue is a duplicate of issue #3288 which has been fixed. See also issue #3194 and issue #3319 which resulted in further related improvements to osmo-ggsn's IPCP parsing.

IIRC the root cause for this was a component not starting and the main test controller failing to notice.
Like, say, a config file format has changed and ggsn(??) fails to start up, then the ttcn3-ggsn-test should timeout on that and not stay stuck forever.

I have found the problem. If osmo-ggsn is not running, the first test runs into a connection failure on the VTY TELNET port:

17:47:29.140113 mtc GGSN_Tests.ttcn:100 TELNET test port (GGSNVTY): GGSNVTY: Try to connect
17:47:29.140165 mtc GGSN_Tests.ttcn:100 Dynamic test case error: Error connecting 127.0.0.1 (Connection refused)
17:47:29.140218 mtc GGSN_Tests.ttcn:100 setverdict(error): none -> error

and then the second test gets stuck forever on this line in f_vty_init():

map(self:GGSNVTY, system:GGSNVTY);

I have traced the hang into the TELNET port, and I found a setting we can use to avoid this issue.
See https://gerrit.osmocom.org/#/c/osmo-ttcn3-hacks/+/10181

An alternative solution might be to reset the port from TTCN3 code but I am unsure how that could be accomplished.
There is C++ code which will reset the port but I don't know whether that code is reachable from TTCN3.

Note that this problem applies to all TTCN3 test suites, not just GGSN.
Apparently, the reason this problem was first observed with GGSN test is that the VTY port is the first port these tests try to use.
Whereas e.g. the BTS tests will only try to open the VTY once some other communication with the osmo-bts process has already succeeded.

Above patch has been merged.

Patch has been reverted because docker ttcn3-bsc-tests were failing with "Verdict: fail reason: VTY Timeout for prompt"
https://gerrit.osmocom.org/c/osmo-ttcn3-hacks/+/10187

I have found the problem which caused BSC tests to fail.

See https://gerrit.osmocom.org/c/osmo-ttcn3-hacks/+/10196 for an updated patch.

Closing this again since the above patch has been merged and we're not seeing any fallout this time.