Feature #3089
closed
osmo-ttcn3-hacks: plainly show changes in test results / show expected-to-fail tests
Added by neels about 6 years ago.
Updated about 6 years ago.
Description
When running TTCN3 tests manually, it can be hard to see whether a code change has caused test results to change.
One part is to have a nicely formatted summary of the test case names and their respective results.
The other part is to know which tests are currently known to fail ("XFAIL").
Seeing the individual test results can be done by looking at the junit XML output. Its human readability is slightly limited, but fairly ok.
An XFAIL mechanism would be nice to add somehow. Another idea is to group the tests in two sections, one that we currently expect to pass, and the other that we currently know to fail, so that it is easy to spot whether something was broken or fixed.
python isn't installed in the docker images though, doubtful that we want to add that for some string parsing.
the result of the summary logger looks like this:
(not sure why this run had so many fails, but that's unrelated)
▶ cat summary-686.log
BSC_Tests.TC_ctrl_msc_connection_status: FAIL
BSC_Tests.TC_ctrl_msc0_connection_status: FAIL
BSC_Tests.TC_ctrl: FAIL
BSC_Tests.TC_chan_act_noreply: FAIL
BSC_Tests.TC_chan_act_counter: FAIL
BSC_Tests.TC_chan_act_ack_noest: FAIL
BSC_Tests.TC_chan_act_ack_est_ind_noreply: FAIL
BSC_Tests.TC_chan_act_ack_est_ind_refused: FAIL
BSC_Tests.TC_chan_act_nack: FAIL
BSC_Tests.TC_chan_exhaustion: FAIL
BSC_Tests.TC_chan_rel_rll_rel_ind: FAIL
BSC_Tests.TC_chan_rel_conn_fail: FAIL
BSC_Tests.TC_chan_rel_hard_clear: FAIL
BSC_Tests.TC_chan_rel_hard_rlsd: FAIL
BSC_Tests.TC_chan_rel_a_reset: FAIL
BSC_Tests.TC_outbound_connect: FAIL
BSC_Tests.TC_assignment_cic_only: FAIL
BSC_Tests.TC_assignment_csd: FAIL
BSC_Tests.TC_assignment_ctm: FAIL
BSC_Tests.TC_assignment_sign: FAIL
BSC_Tests.TC_assignment_fr_a5_0: pass
BSC_Tests.TC_assignment_fr_a5_1: pass
BSC_Tests.TC_assignment_fr_a5_1_codec_missing: pass
BSC_Tests.TC_assignment_fr_a5_3: pass
BSC_Tests.TC_assignment_fr_a5_4: pass
BSC_Tests.TC_rll_est_ind_inact_lchan: FAIL
BSC_Tests.TC_rll_est_ind_inval_sapi1: FAIL
BSC_Tests.TC_rll_est_ind_inval_sapi3: FAIL
BSC_Tests.TC_rll_est_ind_inval_sacch: FAIL
BSC_Tests.TC_paging_imsi_nochan: FAIL
BSC_Tests.TC_paging_tmsi_nochan: FAIL
BSC_Tests.TC_paging_tmsi_any: FAIL
BSC_Tests.TC_paging_tmsi_sdcch: FAIL
BSC_Tests.TC_paging_tmsi_tch_f: FAIL
BSC_Tests.TC_paging_tmsi_tch_hf: FAIL
BSC_Tests.TC_paging_imsi_nochan_cgi: FAIL
BSC_Tests.TC_paging_imsi_nochan_lac_ci: FAIL
BSC_Tests.TC_paging_imsi_nochan_ci: FAIL
BSC_Tests.TC_paging_imsi_nochan_lai: FAIL
BSC_Tests.TC_paging_imsi_nochan_lac: FAIL
BSC_Tests.TC_paging_imsi_nochan_all: FAIL
BSC_Tests.TC_paging_imsi_nochan_plmn_lac_rnc: FAIL
BSC_Tests.TC_paging_imsi_nochan_rnc: FAIL
BSC_Tests.TC_paging_imsi_nochan_lac_rnc: FAIL
BSC_Tests.TC_paging_imsi_nochan_lacs: FAIL
BSC_Tests.TC_paging_imsi_nochan_lacs_empty: FAIL
BSC_Tests.TC_paging_imsi_a_reset: FAIL
BSC_Tests.TC_paging_imsi_load: FAIL
BSC_Tests.TC_paging_counter: FAIL
BSC_Tests.TC_rsl_drop_counter: FAIL
BSC_Tests.TC_rsl_unknown_unit_id: pass
BSC_Tests.TC_oml_unknown_unit_id: pass
BSC_Tests.TC_classmark: pass
BSC_Tests.TC_unsol_ass_fail: pass
BSC_Tests.TC_unsol_ass_compl: pass
BSC_Tests.TC_unsol_ho_fail: pass
BSC_Tests.TC_err_82_short_msg: pass
BSC_Tests.TC_err_84_unknown_msg: pass
BSC_Tests.TC_ho_int: pass
BSC_Tests.TC_bssap_rlsd_does_not_cause_bssmap_reset: FAIL
BSC_Tests.TC_bssmap_clear_does_not_cause_bssmap_reset: FAIL
BSC_Tests.TC_ms_rel_ind_does_not_cause_bssmap_reset: FAIL
So whichever way we obtain such a summary, we could keep one around as the currently expected result,
and make the jenkins ttcn3 job pass as long as no previously passing tests started to fail.
I would commit such a summary of the expected results, but a different idea: an easy way
to make an unexpected pass stick around for next time could be to keep it in the jenkins workspace...
(or both)
Just ideas so far.
- Has duplicate Feature #3136: osmo-ttcn3-hacks: provide a way to summarize test results in a human readable way, and to define currently expected test results added
- Status changed from New to Closed
created a duplicate by accident. Let's go on there. #3136
Also available in: Atom
PDF