Feature #3089
closedosmo-ttcn3-hacks: plainly show changes in test results / show expected-to-fail tests
0%
Description
When running TTCN3 tests manually, it can be hard to see whether a code change has caused test results to change.
One part is to have a nicely formatted summary of the test case names and their respective results.
The other part is to know which tests are currently known to fail ("XFAIL").
Seeing the individual test results can be done by looking at the junit XML output. Its human readability is slightly limited, but fairly ok.
An XFAIL mechanism would be nice to add somehow. Another idea is to group the tests in two sections, one that we currently expect to pass, and the other that we currently know to fail, so that it is easy to spot whether something was broken or fixed.
Related issues
Updated by neels about 6 years ago
I played around for a bit with https://github.com/eclipse/titan.core/tree/master/loggerplugins/JUnitLogger2
and got a hacky summary logger cooked up here: https://git.osmocom.org/osmo-ttcn3-hacks/log/?h=neels/summary_logger
It works, but installation is a bit odd. It needs to be compiled and installed as a library .so file,
I copied static makefiles from the titan core repository to get it installed.
Then it needs to be added as a logging module in Common.cfg.
Don't know if that's really worth using: I guess a quick py or grep/sed script to parse the junit xml instead would get us less complications and similar results.
Updated by neels about 6 years ago
python isn't installed in the docker images though, doubtful that we want to add that for some string parsing.
Updated by neels about 6 years ago
the result of the summary logger looks like this:
(not sure why this run had so many fails, but that's unrelated)
▶ cat summary-686.log BSC_Tests.TC_ctrl_msc_connection_status: FAIL BSC_Tests.TC_ctrl_msc0_connection_status: FAIL BSC_Tests.TC_ctrl: FAIL BSC_Tests.TC_chan_act_noreply: FAIL BSC_Tests.TC_chan_act_counter: FAIL BSC_Tests.TC_chan_act_ack_noest: FAIL BSC_Tests.TC_chan_act_ack_est_ind_noreply: FAIL BSC_Tests.TC_chan_act_ack_est_ind_refused: FAIL BSC_Tests.TC_chan_act_nack: FAIL BSC_Tests.TC_chan_exhaustion: FAIL BSC_Tests.TC_chan_rel_rll_rel_ind: FAIL BSC_Tests.TC_chan_rel_conn_fail: FAIL BSC_Tests.TC_chan_rel_hard_clear: FAIL BSC_Tests.TC_chan_rel_hard_rlsd: FAIL BSC_Tests.TC_chan_rel_a_reset: FAIL BSC_Tests.TC_outbound_connect: FAIL BSC_Tests.TC_assignment_cic_only: FAIL BSC_Tests.TC_assignment_csd: FAIL BSC_Tests.TC_assignment_ctm: FAIL BSC_Tests.TC_assignment_sign: FAIL BSC_Tests.TC_assignment_fr_a5_0: pass BSC_Tests.TC_assignment_fr_a5_1: pass BSC_Tests.TC_assignment_fr_a5_1_codec_missing: pass BSC_Tests.TC_assignment_fr_a5_3: pass BSC_Tests.TC_assignment_fr_a5_4: pass BSC_Tests.TC_rll_est_ind_inact_lchan: FAIL BSC_Tests.TC_rll_est_ind_inval_sapi1: FAIL BSC_Tests.TC_rll_est_ind_inval_sapi3: FAIL BSC_Tests.TC_rll_est_ind_inval_sacch: FAIL BSC_Tests.TC_paging_imsi_nochan: FAIL BSC_Tests.TC_paging_tmsi_nochan: FAIL BSC_Tests.TC_paging_tmsi_any: FAIL BSC_Tests.TC_paging_tmsi_sdcch: FAIL BSC_Tests.TC_paging_tmsi_tch_f: FAIL BSC_Tests.TC_paging_tmsi_tch_hf: FAIL BSC_Tests.TC_paging_imsi_nochan_cgi: FAIL BSC_Tests.TC_paging_imsi_nochan_lac_ci: FAIL BSC_Tests.TC_paging_imsi_nochan_ci: FAIL BSC_Tests.TC_paging_imsi_nochan_lai: FAIL BSC_Tests.TC_paging_imsi_nochan_lac: FAIL BSC_Tests.TC_paging_imsi_nochan_all: FAIL BSC_Tests.TC_paging_imsi_nochan_plmn_lac_rnc: FAIL BSC_Tests.TC_paging_imsi_nochan_rnc: FAIL BSC_Tests.TC_paging_imsi_nochan_lac_rnc: FAIL BSC_Tests.TC_paging_imsi_nochan_lacs: FAIL BSC_Tests.TC_paging_imsi_nochan_lacs_empty: FAIL BSC_Tests.TC_paging_imsi_a_reset: FAIL BSC_Tests.TC_paging_imsi_load: FAIL BSC_Tests.TC_paging_counter: FAIL BSC_Tests.TC_rsl_drop_counter: FAIL BSC_Tests.TC_rsl_unknown_unit_id: pass BSC_Tests.TC_oml_unknown_unit_id: pass BSC_Tests.TC_classmark: pass BSC_Tests.TC_unsol_ass_fail: pass BSC_Tests.TC_unsol_ass_compl: pass BSC_Tests.TC_unsol_ho_fail: pass BSC_Tests.TC_err_82_short_msg: pass BSC_Tests.TC_err_84_unknown_msg: pass BSC_Tests.TC_ho_int: pass BSC_Tests.TC_bssap_rlsd_does_not_cause_bssmap_reset: FAIL BSC_Tests.TC_bssmap_clear_does_not_cause_bssmap_reset: FAIL BSC_Tests.TC_ms_rel_ind_does_not_cause_bssmap_reset: FAIL
So whichever way we obtain such a summary, we could keep one around as the currently expected result,
and make the jenkins ttcn3 job pass as long as no previously passing tests started to fail.
I would commit such a summary of the expected results, but a different idea: an easy way
to make an unexpected pass stick around for next time could be to keep it in the jenkins workspace...
(or both)
Just ideas so far.
Updated by neels about 6 years ago
- Has duplicate Feature #3136: osmo-ttcn3-hacks: provide a way to summarize test results in a human readable way, and to define currently expected test results added
Updated by neels about 6 years ago
- Status changed from New to Closed
created a duplicate by accident. Let's go on there. #3136
