https://projects.osmocom.org/https://projects.osmocom.org/favicon.ico?16647414092018-09-30T08:16:57ZOpen Source Mobile CommunicationsCellular Network Infrastructure - Feature #2385: validate debian rules/control as part of jenkins build testinghttps://projects.osmocom.org/issues/2385?journal_id=116972018-09-30T08:16:57Zlaforge
<ul><li><strong>Assignee</strong> set to <i>osmith</i></li><li><strong>Priority</strong> changed from <i>Normal</i> to <i>Low</i></li></ul> Cellular Network Infrastructure - Feature #2385: validate debian rules/control as part of jenkins build testinghttps://projects.osmocom.org/issues/2385?journal_id=165792019-11-29T12:31:54Zosmith
<ul><li><strong>% Done</strong> changed from <i>0</i> to <i>10</i></li></ul><p>I've done some testing and research to figure out the best way to implement this.</p>
Caveats:
<ul>
<li>we must also build debian packages of all dependencies, and install them beforehand</li>
<li>ideally we would build the debian packages in a fresh debian docker container, and use "apt-get build-dep ." </li>
<li>building all dependencies and the project that was modified itself will take just as long as the current build job</li>
</ul>
<p>So this is more than just a simple change in each project's jenkins.sh script.</p>
<p>Since this takes at least as long as running the current jenkins.sh, I think we should run it in parallel to jenkins.sh.</p>
<p>It would also be nice if we did not have to modify each repository to implement this, so I'd put a common script in osmo-ci.git instead.</p>
Concept:
<ul>
<li>create a new script in osmo-ci.git, which:
<ul>
<li>builds debian packages for all dependencies of a program, each in an empty docker container and by installing dependencies with "apt-get build-dep" </li>
<li>already built dependencies get installed in that empty docker container first</li>
<li>finally build the debian package for the project that we are currently verifying</li>
</ul>
</li>
<li>adjust gerrit-verifications.yml and master-builds.yml to run this script in parallel to the existing jenkins.sh for the projects where we want to test this (using build axes?)</li>
</ul>
<p>This is not as trivial as I thought, so I will not continue with this issue right now.</p> Cellular Network Infrastructure - Feature #2385: validate debian rules/control as part of jenkins build testinghttps://projects.osmocom.org/issues/2385?journal_id=165812019-11-29T14:30:11Zlaforge
<ul></ul><p>What about using the nightly debian package feed for all the dependencies? Yes, that<br />poses some kind of danger during a day, (i.e. libosmocore patch is merged, and one hour later<br />somebody wants to verify a patch using that feature in osmo-msc), but maybe that's acceptable?</p>
<p>In order to improve on that, we could also be something like a "master" package feed which rebuilds<br />packages as soon as a commit is made to master. This could probably be done using OBS, as rebuilding a<br />single package after a commit to our master (e.g. libosmocore.git) should be relatively quick compared to the<br />full-rebuild-everything on nightly? If OBS is not suitable, we could also have a different mechanism (like<br />pbuilder), but I would argue for trying OBS first. At least it should get the propagation delay from<br />"libosmocore patch was merged" from one day (nightly feed) to some (tens of) minutes.</p>
<p>This way we avoid rebuilding all dependencies all the time, which looks like a rather big waste<br />of time (and energy) to me.</p> Cellular Network Infrastructure - Feature #2385: validate debian rules/control as part of jenkins build testinghttps://projects.osmocom.org/issues/2385?journal_id=172382020-01-17T13:20:31Zosmith
<ul></ul><blockquote>
<p>This way we avoid rebuilding all dependencies all the time, which looks like a rather big waste<br />of time (and energy) to me.</p>
</blockquote>
<p>Indeed. I like the idea with the "master" package feed, using OBS. If a patch depends on a patch from another repo that was just merged, waiting ~10 minutes and manually re-triggering the jenkins verification seems like a very good trade-off to me.</p>
<p>(I would like to work on this, but since this has "Priority: Low" I won't start it now.)</p> Cellular Network Infrastructure - Feature #2385: validate debian rules/control as part of jenkins build testinghttps://projects.osmocom.org/issues/2385?journal_id=211832021-02-06T13:58:04Zlaforge
<ul><li><strong>Priority</strong> changed from <i>Low</i> to <i>Normal</i></li></ul> Cellular Network Infrastructure - Feature #2385: validate debian rules/control as part of jenkins build testinghttps://projects.osmocom.org/issues/2385?journal_id=246902022-08-22T09:05:56Zosmith
<ul><li><strong>Related to</strong> <i><a class="issue tracker-1 status-3 priority-3 priority-high3 closed" href="/issues/5654">Bug #5654</a>: OBS errors for libosmo-pfcp, osmo-upf, osmo-hnbgw</i> added</li></ul> Cellular Network Infrastructure - Feature #2385: validate debian rules/control as part of jenkins build testinghttps://projects.osmocom.org/issues/2385?journal_id=249422022-09-21T09:25:04Zosmith
<ul><li><strong>Status</strong> changed from <i>New</i> to <i>In Progress</i></li><li><strong>% Done</strong> changed from <i>10</i> to <i>40</i></li></ul> Cellular Network Infrastructure - Feature #2385: validate debian rules/control as part of jenkins build testinghttps://projects.osmocom.org/issues/2385?journal_id=249452022-09-21T12:04:25Zmsuraev
<ul><li><strong>Related to</strong> <i><a class="issue tracker-1 status-1 priority-2 priority-default" href="/issues/5669">Bug #5669</a>: Test .deb packages built by our OBS</i> added</li></ul> Cellular Network Infrastructure - Feature #2385: validate debian rules/control as part of jenkins build testinghttps://projects.osmocom.org/issues/2385?journal_id=249642022-09-27T14:30:19Zosmith
<ul><li><strong>% Done</strong> changed from <i>40</i> to <i>70</i></li></ul> Cellular Network Infrastructure - Feature #2385: validate debian rules/control as part of jenkins build testinghttps://projects.osmocom.org/issues/2385?journal_id=249972022-10-04T13:12:40Zosmith
<ul><li><strong>% Done</strong> changed from <i>70</i> to <i>90</i></li></ul><p><a class="external" href="https://gerrit.osmocom.org/q/topic:ci-packaging">https://gerrit.osmocom.org/q/topic:ci-packaging</a></p> Cellular Network Infrastructure - Feature #2385: validate debian rules/control as part of jenkins build testinghttps://projects.osmocom.org/issues/2385?journal_id=249982022-10-04T13:29:43Zosmith
<ul></ul>Summary of how it's implemented in the patches linked above:
<ul>
<li>There's a new master repository, where packages get updated as soon as a patch is pushed to master: <a class="external" href="https://obs.osmocom.org/project/show/osmocom:master">https://obs.osmocom.org/project/show/osmocom:master</a></li>
<li>The new <a href="https://jenkins.osmocom.org/jenkins/view/OBS/job/Osmocom_OBS_master_obs.osmocom.org/" class="external">Osmocom_OBS_master</a> job creates the source packages and uploads them to OBS. it runs often, but each run only takes 20-30 seconds as it skips all packages that were not modified.</li>
<li>A docker container with build-essential installed and the master repository configured (and intentionally nothing else to catch missing deps) gets used to install deps and build packages</li>
<li>As it takes forever to install osmo-gsm-manuals-dev + depends, as optimization there's another docker container that has it pre-installed; this gets used if the package has osmo-gsm-manuals-dev in the build depends</li>
<li>I've rolled out the -rpm and -deb verification jobs for osmo-bsc-nat currently for testing (not yet for other repositories, will do after patches are merged): <a class="external" href="https://gerrit.osmocom.org/c/osmo-bsc-nat/+/29492">https://gerrit.osmocom.org/c/osmo-bsc-nat/+/29492</a></li>
</ul> Cellular Network Infrastructure - Feature #2385: validate debian rules/control as part of jenkins build testinghttps://projects.osmocom.org/issues/2385?journal_id=250842022-10-11T13:35:03Zosmith
<ul><li><strong>Status</strong> changed from <i>In Progress</i> to <i>Resolved</i></li><li><strong>% Done</strong> changed from <i>90</i> to <i>100</i></li></ul><p>As noted in code review, my first approach would have lead to lots of comments added to gerrit patches, and thus lots of mails. I've changed it so now a pipeline gets started, and only one comment is written at the end.</p>
<p>The patches are now merged, the jobs are rolled out, and although of course I tested carefully, I'm keeping an eye on the jobs in case there is any fallout.</p>
<p>I've written some documentation here:<br /><a class="external" href="https://osmocom.org/projects/osmocom-servers/wiki/Jenkins_gerrit-verifications_Pipeline">https://osmocom.org/projects/osmocom-servers/wiki/Jenkins_gerrit-verifications_Pipeline</a></p>
<p>And some minor follow-up patches that add back osmo-ci and add a few comments:<br /><a class="external" href="https://gerrit.osmocom.org/q/topic:ci-packaging+status:open">https://gerrit.osmocom.org/q/topic:ci-packaging+status:open</a></p> Cellular Network Infrastructure - Feature #2385: validate debian rules/control as part of jenkins build testinghttps://projects.osmocom.org/issues/2385?journal_id=250862022-10-12T11:44:58Zosmith
<ul><li><strong>% Done</strong> changed from <i>100</i> to <i>90</i></li></ul><p>Setting to 90% again since there are some more fixup patches.</p>
<p><a class="external" href="https://gerrit.osmocom.org/q/topic:ci-packaging+status:open">https://gerrit.osmocom.org/q/topic:ci-packaging+status:open</a></p>
There wasn't much fallout, the only problems I've seen so far and fixed:
<ul>
<li>gerrit-osmo-ttcn3-hacks-build wasn't matched by the regex, thus sending the result comment there didn't work (retriggered the ones manually that were missed)</li>
<li>libosmocore deb/rpm packaging jobs failed on build2-deb11build-ansible in the socket tests, as docker wasn't configured for ipv6 on this node. I've used the ansible rules to deploy the ipv6 config.</li>
<li>the notification for the comment was sent to everybody, not only the owner. turns out we can control that too, and now it only sends a mail to the owner on error, and no mail on success.</li>
</ul> Cellular Network Infrastructure - Feature #2385: validate debian rules/control as part of jenkins build testinghttps://projects.osmocom.org/issues/2385?journal_id=250982022-10-13T11:21:04Zosmith
<ul><li><strong>% Done</strong> changed from <i>90</i> to <i>100</i></li></ul>