Project

General

Profile

Bug #3149

ttcn3-ggsn-tests: can get stuck forever on second test

Added by neels 6 months ago. Updated 2 months ago.

Status:
Resolved
Priority:
Normal
Assignee:
Target version:
-
Start date:
04/09/2018
Due date:
% Done:

0%

Spec Reference:
Tags:

Description

Initializing `JUnitLogger' (v2.0): JUnitLogger writes JUnit-compatible XML
MC@471928bcc090: Configuration file was processed on all HCs.
MC@471928bcc090: Creating MTC on host 471928bcc090.
MC@471928bcc090: MTC is created.
MC2> smtc
Executing all items of [EXECUTE] section.
MC2> MTC@471928bcc090: Starting external command `../ttcn3-tcpdump-start.sh GGSN_Tests.TC_pdp4_act_deact'.
Waiting for tcpdump to start... 0
Waiting for tcpdump to start... 1
MTC@471928bcc090: External command `../ttcn3-tcpdump-start.sh GGSN_Tests.TC_pdp4_act_deact' was executed successfully (exit status: 0).
MTC@471928bcc090: Test case TC_pdp4_act_deact started.
MTC@471928bcc090: GTP1C ConnectionID: 1
MTC@471928bcc090: Dynamic test case error: Error connecting 127.0.0.1 (Connection refused)
MTC@471928bcc090: Test case TC_pdp4_act_deact finished. Verdict: error
MTC@471928bcc090: Starting external command `../ttcn3-tcpdump-stop.sh GGSN_Tests.TC_pdp4_act_deact error'.
Waiting for tcpdump to finish... 0 (prev_count=-1, count=32584)
Waiting for tcpdump to finish... 1 (prev_count=32584, count=32980)
MTC@471928bcc090: External command `../ttcn3-tcpdump-stop.sh GGSN_Tests.TC_pdp4_act_deact error' was executed successfully (exit status: 0).
MTC@471928bcc090: Starting external command `../ttcn3-tcpdump-start.sh GGSN_Tests.TC_pdp4_act_deact_ipcp'.
Waiting for tcpdump to start... 0
Waiting for tcpdump to start... 1
MTC@471928bcc090: External command `../ttcn3-tcpdump-start.sh GGSN_Tests.TC_pdp4_act_deact_ipcp' was executed successfully (exit status: 0).
MTC@471928bcc090: Test case TC_pdp4_act_deact_ipcp started.
MTC@471928bcc090: GTP1C ConnectionID: 1
logs.tgz logs.tgz 13.8 KB ttcn3 logs neels, 04/09/2018 01:04 PM

History

#1 Updated by neels 6 months ago

#3 Updated by laforge 5 months ago

  • Tags set to TTCN3

#4 Updated by laforge 4 months ago

  • Assignee set to stsp

#5 Updated by stsp 3 months ago

I cannot reproduce this problem locally. I have tried to reprocuce it by running the first two GGSN tests in a loop.

I suspect this issue is a duplicate of issue #3288 which has been fixed. See also issue #3194 and issue #3319 which resulted in further related improvements to osmo-ggsn's IPCP parsing.

#6 Updated by stsp 3 months ago

  • Status changed from New to In Progress

#7 Updated by neels 3 months ago

IIRC the root cause for this was a component not starting and the main test controller failing to notice.
Like, say, a config file format has changed and ggsn(??) fails to start up, then the ttcn3-ggsn-test should timeout on that and not stay stuck forever.

#8 Updated by stsp 3 months ago

I have found the problem. If osmo-ggsn is not running, the first test runs into a connection failure on the VTY TELNET port:

17:47:29.140113 mtc GGSN_Tests.ttcn:100 TELNET test port (GGSNVTY): GGSNVTY: Try to connect
17:47:29.140165 mtc GGSN_Tests.ttcn:100 Dynamic test case error: Error connecting 127.0.0.1 (Connection refused)
17:47:29.140218 mtc GGSN_Tests.ttcn:100 setverdict(error): none -> error

and then the second test gets stuck forever on this line in f_vty_init():

map(self:GGSNVTY, system:GGSNVTY);

I have traced the hang into the TELNET port, and I found a setting we can use to avoid this issue.
See https://gerrit.osmocom.org/#/c/osmo-ttcn3-hacks/+/10181

An alternative solution might be to reset the port from TTCN3 code but I am unsure how that could be accomplished.
There is C++ code which will reset the port but I don't know whether that code is reachable from TTCN3.

Note that this problem applies to all TTCN3 test suites, not just GGSN.
Apparently, the reason this problem was first observed with GGSN test is that the VTY port is the first port these tests try to use.
Whereas e.g. the BTS tests will only try to open the VTY once some other communication with the osmo-bts process has already succeeded.

#9 Updated by stsp 3 months ago

  • Status changed from In Progress to Resolved

Above patch has been merged.

#10 Updated by stsp 3 months ago

  • Status changed from Resolved to In Progress

Patch has been reverted because docker ttcn3-bsc-tests were failing with "Verdict: fail reason: VTY Timeout for prompt"
https://gerrit.osmocom.org/c/osmo-ttcn3-hacks/+/10187

#11 Updated by stsp 3 months ago

I have found the problem which caused BSC tests to fail.

See https://gerrit.osmocom.org/c/osmo-ttcn3-hacks/+/10196 for an updated patch.

#12 Updated by stsp 2 months ago

  • Status changed from In Progress to Resolved

Closing this again since the above patch has been merged and we're not seeing any fallout this time.

Also available in: Atom PDF

Add picture from clipboard (Maximum size: 48.8 MB)