Bug #4383
openJenkins build verification is non-deterministic
40%
Description
Recently I submitted a patch that is not supposed to change the unit test output:
https://gerrit.osmocom.org/c/pysim/+/16982/
and it actually does not, but Jenkins verification fails:
https://jenkins.osmocom.org/jenkins/job/gerrit-pysim/261/
Verifying card ... Card contents do not match the test data: Expected: sysmoUSIM-SJS1.ok ------------8<------------ Using PC/SC reader (dev=1) interface Reading ... ICCID: 1122334455667788990 IMSI: 001010000000102 SMSP: ffffffffffffffffffffffffffffffffffffffffffffffffe1ffffffffffffffffffffffff0581005155f5ffffffffffff000000 SPN: Display HPLMN: False Display OPLMN: False PLMNsel: fff11fffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff PLMNwAcT: fff11fffff # MCC: 1651 MNC: 151 AcT: UTRAN, E-UTRAN, GSM, GSM COMPACT, cdma2000 HRPD, cdma2000 1xRTT ffffff0000 # unused ffffff0000 # unused ffffff0000 # unused ffffff0000 # unused ffffff0000 # unused ffffff0000 # unused ffffff0000 # unused ffffff0000 # unused ffffff0000 # unused ffffff0000 # unused ffffff0000 # unused OPLMNwAcT: fff11fffff # MCC: 1651 MNC: 151 AcT: UTRAN, E-UTRAN, GSM, GSM COMPACT, cdma2000 HRPD, cdma2000 1xRTT ffffff0000 # unused ffffff0000 # unused ffffff0000 # unused ffffff0000 # unused ffffff0000 # unused ffffff0000 # unused ffffff0000 # unused ffffff0000 # unused ffffff0000 # unused ffffff0000 # unused ffffff0000 # unused HPLMNAcT: ffffffffff # unused ffffffffff # unused ffffffffff # unused ffffffffff # unused ffffffffff # unused ffffffffff # unused ffffffffff # unused ffffffffff # unused ffffffffff # unused ffffffffff # unused ffffffffff # unused ffffffffff # unused ACC: 0008 MSISDN: Not available AD: 00000002 Done ! ------------8<------------ Got: ------------8<------------ Using PC/SC reader (dev=2) interface Reading ... ICCID: 1122334455667788990 IMSI: 001010000000102 SMSP: ffffffffffffffffffffffffffffffffffffffffffffffffe1ffffffffffffffffffffffff0581005155f5ffffffffffff000000 SPN: Magic <-- The contents of this file do not match the expectations Display HPLMN: True Display OPLMN: True PLMNsel: fff11fffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff PLMNwAcT: fff11fffff # MCC: 1651 MNC: 151 AcT: UTRAN, E-UTRAN, GSM, GSM COMPACT, cdma2000 HRPD, cdma2000 1xRTT ffffff0000 # unused ffffff0000 # unused ffffff0000 # unused ffffff0000 # unused ffffff0000 # unused ffffff0000 # unused ffffff0000 # unused ffffff0000 # unused ffffff0000 # unused ffffff0000 # unused ffffff0000 # unused OPLMNwAcT: fff11fffff # MCC: 1651 MNC: 151 AcT: UTRAN, E-UTRAN, GSM, GSM COMPACT, cdma2000 HRPD, cdma2000 1xRTT ffffff0000 # unused ffffff0000 # unused ffffff0000 # unused ffffff0000 # unused ffffff0000 # unused ffffff0000 # unused ffffff0000 # unused ffffff0000 # unused ffffff0000 # unused ffffff0000 # unused ffffff0000 # unused HPLMNAcT: <-- The contents of this file do not match the expectations fff11fffff # MCC: 1651 MNC: 151 AcT: UTRAN, E-UTRAN, GSM, GSM COMPACT, cdma2000 HRPD, cdma2000 1xRTT ffffff0000 # unused ffffff0000 # unused ffffff0000 # unused ffffff0000 # unused ffffff0000 # unused ffffff0000 # unused ffffff0000 # unused ffffff0000 # unused ffffff0000 # unused ffffff0000 # unused ffffff0000 # unused ACC: 0008 MSISDN: Not available AD: 00000002 Done ! ------------8<------------ Build step 'Execute shell' marked build as failure [WARNINGS]Skipping publisher since build result is FAILURE Finished: FAILURE
As I figured out, this is a side effect of the other changes previously submitted to Gerrit:
https://gerrit.osmocom.org/c/pysim/+/16941
https://gerrit.osmocom.org/c/pysim/+/16972
The problem is that a patch submitted to Gerrit may change the contents of SIM card's file system, so after that Jenkins fails to verify any other changes because the unit test output has changed. In this case both SPN and HPLMNAcT have been changed. Perhaps we need a 'smarter' way (than just comparing stdout and a file) to do build verification, e.g. we could match only those files which we're programming in the unit test. Python has a nice framework for unit tests - https://docs.python.org/3/library/unittest.html, so we could have one unit test per file.
Related issues