Feature #3136
closedosmo-ttcn3-hacks: provide a way to summarize test results in a human readable way, and to define currently expected test results
100%
Description
Use cases:
a) our jenkins builds are continuously failing. We know that certain tests currently fail, so we could mark them expected-to-fail, and mark jenkins job runs as failed only if more failures are added.
b) daily hacking: it is hard to figure out what results the last jenkins run yielded, and it's hard to quickly read the junit xml to compare it to a jenkins result XML. This information should best fall out at the end of a test run, be it in jenkins or manual.
Related issues
Updated by neels about 6 years ago
- Is duplicate of Feature #3089: osmo-ttcn3-hacks: plainly show changes in test results / show expected-to-fail tests added
Updated by neels about 6 years ago
I copied the junit logger code from titan-core and adapted it to write a text file like
/tmp/tmp.I59eae315e/msc-tester/summary-5523.log ----------------------------------------------- pass MSC_Tests.TC_cr_before_reset FAIL MSC_Tests.TC_lu_imsi_reject FAIL MSC_Tests.TC_lu_imsi_timeout_gsup
In the presence of an expected-verdicts.txt file (a copy of a log output like above), the output becomes
/tmp/tmp.I59eae315e/msc-tester/summary-5523.log ----------------------------------------------- pass MSC_Tests.TC_cr_before_reset xfail MSC_Tests.TC_lu_imsi_reject xfail MSC_Tests.TC_lu_imsi_timeout_gsup
In result, we can do something like
if grep -v '^\(pass\|xfail\)' summary*.log; then echo UNEXPECTED RESULTS exit 1 fi
Still need to figure out some details:
- compilation: so far I need to compile the summary logger separately with an adapted Makefile from titan-core, to produce an .so file that is accepted and doesn't segfault.
- ideally make it so that the expected-results.txt can sit in osmo-ttcn3-hacks/foo/ and is used for both dockerized tests and manual tests automagically.
- if some expected-to-fail tests currently start passing, do we still mark the jenkins job as failed? I think we have to remind ourselves to update the expected results somehow.
Jenkins could also keep a separate expected-results file in the workspace and only ensure that we never get worse, and update the file automatically if it gets better.
I'm not actually sure that the logger plugin is the best way to solve this. An external tool that compares, for example, junit xml files would cut short all the linking issues, and we would get a shell return code from it marking failure or success without needing grep magic... The biggest hindrance is that it would be nice not to add huge dependencies like python to the docker images; also the junit xml files don't provide as fine-grained verdicts as ttcn3 outputs (INCONC, ERROR, FAIL all amount to a failure)
Updated by neels about 6 years ago
- Status changed from In Progress to Closed
created a duplicate by accident. Let's go on there. #3136
Updated by neels about 6 years ago
I've got a bash script that parses the junit xml now, which is much less trouble: no compilation, no added dependencies, easy to call on arbitrary paths:
https://gerrit.osmocom.org/7646
Already added the expected results from jenkins runs and integrated in start-testsuite.sh, should be useful for both jenkins runs and manual invocation, now.
On jenkins, we might want to add --allow-xpass (allow unexpected pass of a test), though that is debatable.
If we fixed a test, we should immediately update the expected results. Otherwise the entire scheme becomes futile.
If we wanted to allow xpass, we'd do in docker-playground:
--- a/jenkins-common.sh +++ b/jenkins-common.sh @@ -39,3 +39,5 @@ mkdir $VOL_SUITE_DIR rm -rf $WORKSPACE/logs || /bin/true mkdir -p $WORKSPACE/logs + +export OSMO_TTCN3_COMPARE_ARGS="--allow-xpass"
Updated by neels about 6 years ago
- Status changed from In Progress to Resolved
- % Done changed from 90 to 100
merged