Bug #4383
openJenkins build verification is non-deterministic
40%
Description
Recently I submitted a patch that is not supposed to change the unit test output:
https://gerrit.osmocom.org/c/pysim/+/16982/
and it actually does not, but Jenkins verification fails:
https://jenkins.osmocom.org/jenkins/job/gerrit-pysim/261/
Verifying card ... Card contents do not match the test data: Expected: sysmoUSIM-SJS1.ok ------------8<------------ Using PC/SC reader (dev=1) interface Reading ... ICCID: 1122334455667788990 IMSI: 001010000000102 SMSP: ffffffffffffffffffffffffffffffffffffffffffffffffe1ffffffffffffffffffffffff0581005155f5ffffffffffff000000 SPN: Display HPLMN: False Display OPLMN: False PLMNsel: fff11fffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff PLMNwAcT: fff11fffff # MCC: 1651 MNC: 151 AcT: UTRAN, E-UTRAN, GSM, GSM COMPACT, cdma2000 HRPD, cdma2000 1xRTT ffffff0000 # unused ffffff0000 # unused ffffff0000 # unused ffffff0000 # unused ffffff0000 # unused ffffff0000 # unused ffffff0000 # unused ffffff0000 # unused ffffff0000 # unused ffffff0000 # unused ffffff0000 # unused OPLMNwAcT: fff11fffff # MCC: 1651 MNC: 151 AcT: UTRAN, E-UTRAN, GSM, GSM COMPACT, cdma2000 HRPD, cdma2000 1xRTT ffffff0000 # unused ffffff0000 # unused ffffff0000 # unused ffffff0000 # unused ffffff0000 # unused ffffff0000 # unused ffffff0000 # unused ffffff0000 # unused ffffff0000 # unused ffffff0000 # unused ffffff0000 # unused HPLMNAcT: ffffffffff # unused ffffffffff # unused ffffffffff # unused ffffffffff # unused ffffffffff # unused ffffffffff # unused ffffffffff # unused ffffffffff # unused ffffffffff # unused ffffffffff # unused ffffffffff # unused ffffffffff # unused ACC: 0008 MSISDN: Not available AD: 00000002 Done ! ------------8<------------ Got: ------------8<------------ Using PC/SC reader (dev=2) interface Reading ... ICCID: 1122334455667788990 IMSI: 001010000000102 SMSP: ffffffffffffffffffffffffffffffffffffffffffffffffe1ffffffffffffffffffffffff0581005155f5ffffffffffff000000 SPN: Magic <-- The contents of this file do not match the expectations Display HPLMN: True Display OPLMN: True PLMNsel: fff11fffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff PLMNwAcT: fff11fffff # MCC: 1651 MNC: 151 AcT: UTRAN, E-UTRAN, GSM, GSM COMPACT, cdma2000 HRPD, cdma2000 1xRTT ffffff0000 # unused ffffff0000 # unused ffffff0000 # unused ffffff0000 # unused ffffff0000 # unused ffffff0000 # unused ffffff0000 # unused ffffff0000 # unused ffffff0000 # unused ffffff0000 # unused ffffff0000 # unused OPLMNwAcT: fff11fffff # MCC: 1651 MNC: 151 AcT: UTRAN, E-UTRAN, GSM, GSM COMPACT, cdma2000 HRPD, cdma2000 1xRTT ffffff0000 # unused ffffff0000 # unused ffffff0000 # unused ffffff0000 # unused ffffff0000 # unused ffffff0000 # unused ffffff0000 # unused ffffff0000 # unused ffffff0000 # unused ffffff0000 # unused ffffff0000 # unused HPLMNAcT: <-- The contents of this file do not match the expectations fff11fffff # MCC: 1651 MNC: 151 AcT: UTRAN, E-UTRAN, GSM, GSM COMPACT, cdma2000 HRPD, cdma2000 1xRTT ffffff0000 # unused ffffff0000 # unused ffffff0000 # unused ffffff0000 # unused ffffff0000 # unused ffffff0000 # unused ffffff0000 # unused ffffff0000 # unused ffffff0000 # unused ffffff0000 # unused ffffff0000 # unused ACC: 0008 MSISDN: Not available AD: 00000002 Done ! ------------8<------------ Build step 'Execute shell' marked build as failure [WARNINGS]Skipping publisher since build result is FAILURE Finished: FAILURE
As I figured out, this is a side effect of the other changes previously submitted to Gerrit:
https://gerrit.osmocom.org/c/pysim/+/16941
https://gerrit.osmocom.org/c/pysim/+/16972
The problem is that a patch submitted to Gerrit may change the contents of SIM card's file system, so after that Jenkins fails to verify any other changes because the unit test output has changed. In this case both SPN and HPLMNAcT have been changed. Perhaps we need a 'smarter' way (than just comparing stdout and a file) to do build verification, e.g. we could match only those files which we're programming in the unit test. Python has a nice framework for unit tests - https://docs.python.org/3/library/unittest.html, so we could have one unit test per file.
Related issues
Updated by laforge about 4 years ago
dexter what about this, it has been 3 months? at least a one-line input/opinion from you would be much appreciated.
Updated by dexter about 4 years ago
I had a look at it, first of all the current master of pysim and sysmo-usim-tool test both fine.
From time to time we see problems when sysmo-usim-tool does changes to the file system which then cause the tests of pysim to fail and vice versa. At least in my opinion this wasn't to much of a problem yet, but I agree, the way we test is a bit clumsy.
When I get you right you want to use the APIs provided by pysim to check on the contents of the files and implement those checks as python unit-tests? Maybe we could also make pySim-read.py a bit more scripting friendly. Maybe have commandline arguments to specify which file/parameter to read out. Then we would also have some kind of separation and at the same time we woule improve pySim-read.py so that it can be used from scripts.
Updated by fixeria about 4 years ago
One more change related to this ticket: https://gerrit.osmocom.org/c/pysim/+/18073.
Updated by laforge about 4 years ago
I'm sorry, but this is starting to seriously impede the development of pySim.
From my point of view, what is needed is to properly reset the system to a sane state after executing tests. "system" here includes the contents of the SIM cards.
So if a patch makes some changes to the filesystem, fine - we will then have to re-write the entire filesystem with known data before starting another test. This should work- for user-writable files on all card types
- for ADM1-writable files on all card types
If some files cannot be written even with ADM1, this shouldn't be a problem - it is safe to assume that any [future] patch we test will then also not be able to write those files.
Updated by dexter about 4 years ago
- Status changed from New to In Progress
- % Done changed from 0 to 40
The resetting of the file system contents should take place before the test executes. My idea is to use the erase() method in cards.py. I think at the moment this method is only programmed for one card. What I plan to do is to reset the contents of the files that pysim writes to 0xff, so that they are basically uninitalized. We can do this with a separate tool pySim-erase.py, this way we do not have to touch pySim-prog.py. The erase() method would use functions that do a read-after-write verification of the file contents, so that we can be sure that the card really has been erased. I already have a draft implementation of pySim-erase.py and some methods for the cards class that can do an erase of binary files and records in one line, so filling the erase() method for each card should be easy. For sysmo-isim-sjs2 I already filled the erase() method.
The following patches are in review now:
https://gerrit.osmocom.org/c/pysim/+/18205 cards: remove len calculation
https://gerrit.osmocom.org/c/pysim/+/18206 cards: reset uninitalized EF.AD
https://gerrit.osmocom.org/c/pysim/+/18207 pysim-prog: move ADM sanitation to utils.py
https://gerrit.osmocom.org/c/pysim/+/18208 utils: do not crash when all bytes of EF.IMSI are 0xFF
https://gerrit.osmocom.org/c/pysim/+/18209 commands: add method to determine size of a non record oriented file
https://gerrit.osmocom.org/c/pysim/+/18211 commands: add features to verify data written to files
Updated by laforge about 4 years ago
dexter wrote:
The resetting of the file system contents should take place before the test executes. My idea is to use the erase() method in cards.py. I think at the moment this method is only programmed for one card. What I plan to do is to reset the contents of the files that pysim writes to 0xff, so that they are basically uninitalized.
The problem is that this will probably break once somebody submits code for a new file to gerrit, which is not yet understood by pysim master. This way the card contents changes juts because we tried to verify any random submitted but not yet approved patch.
Hence I think it would be better to really do a full dump / backup of all files on each of the cards once, and then restore that backup before each test run. The "Backup" part is basically what "osmo-sim-test -o" from libosmocore/utils is generating. Maybe all that's needed is the inverse part, i.e. restoring such a file/directory based backup?
We can do this with a separate tool pySim-erase.py, this way we do not have to touch pySim-prog.py.
I agree this test pre-setup should be kept out of pySim-prog. Whether it's C code or python, I don't really have a preference.
The erase() method would use functions that do a read-after-write verification of the file contents, so that we can be sure that the card really has been erased.
If you want to stick with erase rather than restore, you should make sure to generate a list of all files/paths on each of the cards once, and then erase all their contents [assuming one can do this with ADM1 pin on all relevant cards]
Updated by fixeria about 4 years ago
- Related to Feature #4384: Jenkins program-read verification on daily (or weekly) basis added
Updated by fixeria about 3 years ago
The build verification is broken again. It looks like one of the recent pySim patches has changed something on the test card(s):
https://gerrit.osmocom.org/c/pysim/+/23631 Use zero padding for EF['ACC'] field
https://gerrit.osmocom.org/c/pysim/+/23632 Use zero padding for EF['ACC'] field
I tried to upload another patch that does not touch the actual parsing logic, and it fails too:
https://gerrit.osmocom.org/c/pysim/+/23633 Py2 -> Py3: do not inherit classes from object
Updated by dexter about 3 years ago
Hello fixeria
There is indeed something changed on the card:
Administrative data: 00000002
MS operation mode: normal operation
is now:
Administrative data: 80000002
MS operation mode: type approval operations
This is a bit odd, at least the patches I added recently do not touch this field. I will take care of this after lunch.
Updated by dexter about 3 years ago
I have now fixed the problem by resetting EF.AD to its original state.
Updated by laforge almost 2 years ago
- Related to Feature #5127: tool for "re-formatting" [sysmocom] cards in tester added