Cellular Network Infrastructure

h1. OBS Maintenance

{{>toc}}

The Osmocom [[binary packages]] are built with [[OBS]] (Open Build Service) hosted on "obs.osmocom.org":https://obs.osmocom.org. In order to do that, the jenkins jobs "Osmocom_OBS_latest":https://jenkins.osmocom.org/jenkins/job/Osmocom_OBS_latest_obs.osmocom.org/ / "_nightly":https://jenkins.osmocom.org/jenkins/job/Osmocom_OBS_nightly_obs.osmocom.org/ ("config":https://gitea.osmocom.org/osmocom/osmo-ci/src/branch/master/jobs/osmocom-obs.yml) checkout the source repositories and build debian *source* packages, then upload these to OBS.

h2. Install osc

Get the command-line client for managing OBS. When using for the first time, it will ask for the credentials.

<pre>

$ sudo apt install osc

$ osc ls

</pre>

h2. Using openSUSE.org-mirror in meta configs

Each OBS project with packages (e.g. osmocom:nightly, your home: project) depends on other projects for the Linux distributions that the packages should be built for (Debian:12 etc.).

This is defined in the "meta" configuration, e.g. here: https://obs.osmocom.org/projects/osmocom:nightly/meta

In the past we had a direct dependency on openSUSE's OBS instance for these. This had the disadvantage that we couldn't build packages whenever there was an outage at openSUSE's OBS instance.

<pre><code class="xml">

<path project="openSUSE.org:Debian:12" repository="standard"/>

</code></pre>

Nowadays we have a "jenkins job":https://jenkins.osmocom.org/jenkins/view/OBS/job/Osmocom_OBS_sync that syncs the projects from openSUSE's OBS to our OBS once a day. These synchronized projects are under "openSUSE.org-mirror":https://obs.osmocom.org/project/subprojects/openSUSE.org-mirror and can be used as follows:

<pre><code class="xml">

<path project="openSUSE.org-mirror:Debian:12" repository="standard"/>

</code></pre>

Adding or removing distributions is described in [[Add_a_new_distribution_to_OBS|this]] article.

h2. Upgrading specific packages

h3. eclipse-titan

See the article [[upgrading eclipse-titan in the Osmocom OBS]].

h3. nftables / libnftnl

Eventually the nftables version >= 1.0.2 that osmo-upf requires should make it into the distributions we support, at which point we can remove the nftables and libnftnl packages. Until then we build these packages in OBS, this section describes how to upgrad them.

h4. rpm

As of writing, the nftables and libnftnl packages from the Osmocom OBS are linked to openSUSE.org:security:netfilter. When the package gets upgraded there, the build will fail in the Osmocom OBS with:

<pre>

[   29s] error: Bad source: /home/abuild/rpmbuild/SOURCES/libnftnl-1.2.4.tar.bz2: No such file or directory

</pre>

That is because the source tarballs get upgraded, but the spec file has differences and so it does not get upgraded automatically.

Diff the "upstream spec file":https://build.opensuse.org/package/view_file/security:netfilter/libnftnl/libnftnl.spec?expand=1 against the one in the Osmocom OBS and apply the changes, then it should work again.

<pre>

$ $EDITOR /tmp/upstream.spec  # put the upstream spec file here

$ mkdir /tmp/obs

$ cd /tmp/obs

$ osc co osmocom:nightly libnftnl

$ cd osmocom:nightly/libnftnl

$ diff libnftnl.spec /tmp/upstream.spec

$ $EDITOR libnftnl.spec  # apply changes

$ diff libnftnl.spec /tmp/upstream.spec  # verify

$ osc commit

$ rm -rf /tmp/obs

</pre>

Wait until the package has built, then copy it to the other feeds:

<pre>

$ osc copypac osmocom:nightly libnftnl osmocom:latest

$ osc copypac osmocom:nightly libnftnl osmocom:master

</pre>

h4. deb

<pre>

$ mkdir /tmp/obs

$ cd /tmp/obs

$ osc co osmocom:nightly libnftnl-deb

$ cd osmocom:nightly/libnftnl-deb

$ osc del *

</pre>

Download the debian "source package files":https://packages.debian.org/source/sid/libnftnl into the directory and add them.

<pre>

$ osc add *

$ osc commit -m "upgrade to 1.25"

$ rm -rf /tmp/obs

</pre>

Wait until the package has built, then copy it to the other feeds (see rpm).

h2. Workflow for testing OBS configurations

OBS has a "project configuration":https://en.opensuse.org/openSUSE:Build_Service_prjconf, which may need to be adjusted to state preferred dependencies for example. Before we apply project configurations to the official Osmocom OBS projects, we can test them in our own OBS project. Copy the relevant package first (adjust the username accordingly in all following examples):

<pre>

$ osc copypac osmocom:nightly osmo-gsm-manuals home:osmith42

</pre>

Visit the site in your browser:

* https://obs.osmocom.org/package/show/home:osmith42/osmo-gsm-manuals

Set up build targets:

* https://obs.osmocom.org/project/add_repository_from_default_list/home:osmith42

The package should get added to the queue already.

Open your project config next to the osmocom:nightly one:

* https://obs.osmocom.org/project/prjconf/home:osmith42

* https://obs.osmocom.org/project/prjconf/osmocom:nightly

First copy everything from the Osmocom project over to your own project. Then make changes to your config, until the builds are working as expected (dependency problems are resolved etc.). Whenever you change the config, OBS will immediately re-evaluate the build dependencies, no need to upload the source packages again.

When everything works as expected, copy the config changes you have made to the Osmocom project (nightly and latest).

Note that changing osmocom's project config did not seem to have any effect. It seems that we really must change both @osmocom:nightly@ and @osmocom:latest@ instead.

h2. Workflow for modifying packages

h3. (Optional) modify the debian dir

* clone the source git repository

* make changes to the debian dir

* commit your changes

* push to a "user/..." branch

h3. Build source packages and upload to your own OBS project

* clone osmo-ci.git

* open "osmo-nightly-packages.sh":https://gitea.osmocom.org/osmocom/osmo-ci/src/branch/master/scripts/osmocom-nightly-packages.sh

* if you intend to make changes to the script, do them and commit the changes

* change "PROJ":https://git.osmocom.org/osmo-ci/tree/scripts/osmocom-nightly-packages.sh?id=dc5fc51c1fa76e64db774ecb9e71fc25ad6c46cd#n10 to home:USERNAME

* if you made changes to the debian dir, insert your branch name after the related "checkout line":https://git.osmocom.org/osmo-ci/tree/scripts/osmocom-nightly-packages.sh?id=dc5fc51c1fa76e64db774ecb9e71fc25ad6c46cd#n172 (e.g. <code>checkout osmo-gsm-manuals osmith/some-random-change</code>)

* run osmocom-nightly-packages.sh

* check in your browser if all packages have been built by OBS as expected (iterate until you get everything right)

When everything works fine, contribute your patches with [[Gerrit]] as usually.

h2. Troubleshooting

h3. A script must run before building the source package (to download dependencies etc.)

If we have control over the git repository, we can add a <code>contrib/generate_build_dep.sh</code> script to it ("example":https://gitea.osmocom.org/osmith/osmo_dia2gsup/src/branch/master/contrib/generate_build_dep.sh). It will be executed by the OBS scripts when building source packages.

Otherwise add a <code>prepare_project_</code> function to <code>scripts/obs/lib/srcpkg.py</code> in <code>osmo-ci.git</code>. See the existing functions there for reference.

h3. Source tarball not found

If a build suddenly fails with an error like this:

<pre>

[   49s] error: File /home/abuild/rpmbuild/SOURCES/zeromq-4.3.4.tar.gz: No such file or directory

</pre>

Then it is probably linked to an upstream package from openSUSE OBS and needs to be adjusted. We have it set up like this to automatically get new versions if they are packaged upstream. However the downside is that oftentimes we need to adjust the spec file, which means that it can't automatically be updated. Therefore the new source tarball gets added (in the example zeromq-4.3.5.tar.gz), but the spec file still mentions the old version.

How to fix it:

* Follow the links on our OBS to the upstream project ("example":https://build.opensuse.org/package/show/openSUSE:Factory/zeromq)

* Look at the diff of the upgrade ("example":https://build.opensuse.org/request/show/1119932)

* Apply the changes to our spec files (usually in osmocom:latest, osmocom:nightly, osmocom:master)

h3. Package is not resolvable

<pre>

have choice for jadetex needed by docbook-utils: jadetex texlive-formats-extra

</pre>

You'll need to adjust the OBS project configuration (mind the testing workflow above):

<pre>

Prefer: texlive-htmlxml

</pre>

h3. Reproduce the nightly builds locally

<pre>

$ osc co

$ osc build

</pre>

h2. E-Mail Notifications of build failures

Build failure notifications are sent to the "gerrit-log mailing list":https://lists.osmocom.org/mailman/listinfo/gerrit-log

h2. Remove a package from a binary repository

When a package can't be built for e.g. debian 10 anymore, an old version of the package will still remain in the binary repository. A workaround is to delete the package and undelete it again:

<pre>

$ osc rdelete osmocom:nightly osmocom-bb

$ osc undelete osmocom:nightly osmocom-bb

</pre>

h2. See also

* [[Linux distributions]]: which distributions we support for how long

* [[Add a new distribution to OBS]]

* https://en.opensuse.org/Portal:Build_Service

* https://en.opensuse.org/openSUSE:Build_Service_prjconf