Bug #5802
closeddon't globally block/conflict TTCN3 jobs; just don't run same job on same node multiple times
100%
Description
We have jobs, particularly the ttcn3 dockerized test jobs, which can not run multiple times on the same node as they use static IP addresses and docker refuses to have the same addresses in multiple isolated networks/namespaces.
Right now the "workaround" is to globally block the job. So if any of our TTCN3 bts tests (e.g. master) is running anywhere, we won't do another BTS test (e.g. latest, centos) on any other executor. Particularly since those BTS/BSC tests are taking hours to execute, this is causing significant delays.
There are multiple suggestions in
https://stackoverflow.com/questions/36454130/how-do-i-prevent-two-pipeline-jenkins-jobs-of-the-same-type-to-run-in-parallel-o how to achieve the desired behavior. I of course don't know if any of them work, but it might be worth having a look.
Updated by laforge over 1 year ago
it looks like https://plugins.jenkins.io/throttle-concurrents/ could do the trick: throttling the number of concurrend builds ofa project running per node...
Updated by osmith over 1 year ago
- Status changed from New to In Progress
- % Done changed from 0 to 70
docker refuses to have the same addresses in multiple isolated networks/namespaces.
I've changed the logic in the jenkins.sh scripts to automatically find a free subnet. This way blocking shouldn't be necessary anymore.
Patches: https://gerrit.osmocom.org/q/topic:ttcn3-subnet
(I'll wait with rolling this out until after the holidays, as this is a major change.)
Updated by osmith over 1 year ago
There's some fallout, multiple ttcn3 jobs are failing now. Looking into it.
Updated by osmith over 1 year ago
- % Done changed from 80 to 90
Updated by osmith over 1 year ago
After the fixups, looks like all testsuites are running properly again. I've restarted them and removed the failed entry of today.
One more patch to increase a timeout that failed: https://gerrit.osmocom.org/c/docker-playground/+/31001
Updated by osmith over 1 year ago
- Status changed from In Progress to Resolved
- % Done changed from 90 to 100