Feature #4384
openJenkins program-read verification on daily (or weekly) basis
0%
Description
We already have build verification for some projects running once per day:
https://jenkins.osmocom.org/jenkins/view/master/
It would be nice to have daily or at least weekly (given the low amount of contributions) master build verification for pySim too.
Related issues
Updated by laforge about 4 years ago
- Status changed from New to Feedback
- Assignee changed from dexter to fixeria
fixeria what would be "build verifcation" of a python program? It's not like we're compiling anything...
Updated by fixeria about 4 years ago
- Related to Bug #4383: Jenkins build verification is non-deterministic added
Updated by fixeria about 4 years ago
- Subject changed from Jenkins build verification on daily (or weekly) basis to Jenkins program-read verification on daily (or weekly) basis
The idea to have a regular testing was proposed in the IRC while discussing #4383. If I remember correctly, back then Jenkins verification was broken (some EF was changed) by a change submitted to Gerrit, and this remained undiscovered for some time. So if we had automatic daily or weekly verification, the problem could be spotted and fixed earlier. Right now one would need to walk over the list of all changes trying to find the culprit.
Updated by laforge over 3 years ago
fixeria wrote:
I'm not really following you here. What happened:The idea to have a regular testing was proposed in the IRC while discussing #4383. If I remember correctly, back then Jenkins verification was broken (some EF was changed) by a change submitted to Gerrit, and this remained undiscovered for some time. So if we had automatic daily or weekly verification, the problem could be spotted and fixed earlier. Right now one would need to walk over the list of all changes trying to find the culprit.
- a patch was submitted to jenkins, changing some unexpected data on the card
- all follow-up verifications failed due to the wrong contents on the card
- completely re-format the cards between each two test runs. (re-install the OS, card profile, dynamic data)
- is only possible for sysmocom cards, as we don't have the kind of data/tools/docs for others
- would introduce a lot of unusual flash wear, so I'd assume the cards would not survive all too long. Once again not a problem for sysmocom cards which we have thousands, but others we only have few
- keep a backup of all "well-known" (3GPP standard files) of each card, and verify that at start-up. If any files differ, re-write selectively only those files from the backup
- would work with probably almost any card (maybe except the super cheap magicSIM that may not permit regular write operations to all files?
- would not help in case of unexpected modifications were made to non-standard/proprietary files that we cannot backup/restore in a generic way
- keep a completely separate physical test setip with different card readers + cards for testing master and jenkins. Then any breakage introduced by random jenkins patches would not render us unable to see if master still works
- '1' is out of the question
- '2' is doable, I think I could hack up a backup-and-restore tool relatively quickl (for SIM/USIM/ISIM standard files)
- '3' sounds like it's not a lot of effort, physically. We could even use one of our sysmoOCTSIM now. However, I don'y think we can use the exact same test expectations in both setups, as the card ICCID will be different, and possibly also ADM pin, etc.
dexter, any input to this?