Miscellaneous Projects » Core testing infrastructure

So it somehow seems that if a longer series of patches is committed to libosmocore, each of them gets built as OBS packge for each arch/distibution, and that's what takes so long? Can we somehow get it to only build the most recent commit at a time, and first build that all the way to the end and make sure those packages are published/available for further downstream builds?

From what I have observed, it works like this:

• new source package gets pushed to OBS for libosmocore
• OBS starts building libosmocore, and queues all reverse depends to be rebuilt afterwards
• when the libosmocore package is done, it does not yet get published because that would render the repository in an inconsistent state - the other packages are not rebuilt yet against this version
• OBS continues building packages in the queue
• when a new source package gets pushed to OBS before the queue is done, it repeats with building libosmocore again. the packages are not published until the whole queue is done once for one distribution/arch combination

It's not exactly a rare situation that we add something to one library, and then there are also patches that use those additions elsewhere.

For some more context, I've attached the graph of the previous days. At least it does not seem to happen every day that it gets stuck like this.

Can we somehow get it to only build the most recent commit at a time, and first build that all the way to the end and make sure those packages are published/available for further downstream builds?

I guess in theory we could check whether OBS is currently building source packages, and in that case, not send further source packages until it is completely done. In some scenarios it will be faster until you have libosmocore built with a patch you need, but in others it will be slower. And the total number of packages built will increase.

faster case:

• push libosmocore patch 1
• reverse deps get rebuilt (queue 1)
• before queue 1 is done, libosmocore patch 2 gets pushed, but we don't push a new source package to OBS yet because queue 1 is not done
• queue 1 is complete
• we push libosmocore patch 2
• queue 2 starts building
• osmo-bsc patch that needs libosmocore patch 1 gets verified in gerrit
• it can already be verified, because we don't need to wait for queue 2

slower case:

• push libosmocore patch 1
• reverse deps get rebuilt (queue 1)
• before queue 1 is done, libosmocore patch 2 gets pushed, but we don't push a new source package to OBS yet because queue 1 is not done
• queue 1 is complete
• we push libosmocore patch 2
• queue 2 starts building
• osmo-bsc patch that needs libosmocore patch 2 gets verified in gerrit
• it fails, because queue 2 is not done yet

so all in all, it will cause more package builds, it may be slower depending on luck / what you need to build against, and it makes the logic for pushing packages a bit more complicated... IMHO we should try to optimize other things instead.

we can make find some other way to make sure the (in this case) libosmocore package is somehow available for gerrit build verification after it has been built, and not only after the entire universe has been re-built?

I guess this would require a hack of extracting the not-yet-published package from OBS somehow, and installing that instead of the real repository. IMHO too much effort.

we can make sure that repos/feeds are updated after building [only] the debian10/debian12 on amd64 which we use for gerrit build verification [and even prioritize those]?

As I understand it, the feeds do get updated after one distro + arch combination is complete. However OBS builds multiple distros in parallel, so they more or less finish at the same time.

We have built osmocom:master for the following distros, all x86_64 only:

• Debian_10
• Debian_11
• Debian_12
• CentOS_8
• openSUSE_Tumbleweed

I just removed CentOS_8 and openSUSE_Tumbleweed. We don't use them in CI verification (CentOS8 not anymore since recently), so that should already speed it up significantly.

(We do use debian 10, 11 and 12. Most projects build test against 10 and 12, but some that don't work on 10 test against 11 and 12.)

Another thing to look into could be adding additional workers. I think it would be good to add another 8 workers on build5 once we have set it up (#6186).

And yet another thing could be making the package builds faster, e.g. by pre-installing dependencies for osmo-gsm-manuals... I think it is possible to build in a pre-built image, but haven't looked into this in detail yet.