Cellular Network Infrastructure

Assigning this to myself, because another issue that is assigned to me (#3386) depends on this one.

The idea is to keep the common parts and build related scripts (makefile, etc) in osmo-gsm-manuals.git, and then conditionally enable building them by using a configure flag (eg. --enable-man --with-man-path=/path/to/osmo-gsm-manuals.git/).

pespin: I wonder if that works properly with packaging the man pages for Debian later on. We could also extend osmo-gsm-manuals.git with a "make install" target and write a .pc file, that points to the shared code (pkg-config allows custom variables). Then we could create a "osmo-gsm-manuals-dev" package as build time dependency for the other osmo packages. What do you think?

Fine for me. Keep in mind that it may be interesting to have some Makefile installed by "make install" in osmo-gsm-manuals.git and reachable through pkg-config custom variable which can then be included as described in initial post. This way we avoid copy&paste same stuff on how to build adoc stuff into each project. Up to you to find best solution.

I have researched how this issue relates to the Jenkins jobs.

• push to master:
  • osmo-gsm-manuals.git - jenkins job "master-osmo-gsm-manuals" (from master-builds.yml)
    • calls ./contrib/jenkins.sh --publish
    • builds the manuals for all projects
    • runs rsync to publish the "./out" folder
• push to gerrit:
  • osmo-gsm-manuals.git - jenkins job "gerrit-osmo-gsm-manuals" (from gerrit-verifications.yml)
    • calls ./contrib/jenkins.sh
    • builds the manuals for all projects

• push to master:
  • osmo-gsm-manuals.git
    • build and publish the documentation of all projects
  • osmo-*.git
    • build and publish the documentation of that project
• push to gerrit:
  • osmo-gsm-manuals.git
    • build the documentation of all projects
  • osmo-*.git
    • build the repo with its documentation

Thanks for the feedback!

Time for a status update. I've pushed my WIP code to osmith/move-manuals-to-project-repos branches in both osmo-msc.git and osmo-gsm-manuals.git. My plan is to make it work with MSC first (as proof of concept), then with all other projects.

• moved osmo-gsm-manuals.git/OsmoMSC to osmo-msc.git/doc/man
• osmo-gsm-manuals.git installs the common files to $prefix/share/osmo-gsm-manuals (e.g. /usr/local/share/osmo-gsm-manuals):
  

  $ autoreconf -fi
  $ ./configure
  $ make
  $ make install
  

• a pkg-config file gets generated, with a custom "topdir" variable that points to $prefix/share/osmo-gsm-manuals
• the new topdir variable is queried from pkg-config and used by the project's manual build Makefile.am instead of the old "TOPDIR := .."
• "../build", "../common" was hardcoded in a lot of places, this was changed to use that variable where possible (asciidoc), or adjusted to use a symlink to $topdir/common (doxygen)
• osmo-msc.git's pdf is building now (needs more polishing though, images are probably missing etc.)
• osmo-msc.git builds the pdf only when --enable-man is specified at the configure line
• the check for installed dependencies was moved from osmo-gsm-manuals.git's global Makefile to "osmo-gsm-manuals-check-depends" installed to $prefix/bin. It gets called inside osmo-msc.git's configure when --enable-man is set.
• I have moved libosmocore.git/doc/vty/merge_doc.xsl to osmo-gsm-manuals.git/merge_doc.xsl. It was not used anywhere in libosmocore, but it is needed for building the manuals.

Also I've noticed that GIT_VERSION and GIT_DATE are problematic here:
https://git.osmocom.org/osmo-gsm-manuals/tree/build/Makefile.asciidoc.inc

GIT_VERSION := $(shell git describe --abbrev=4 --dirty --always --tags)
GIT_DATE := $(shell $(TOPDIR)/build/unix-time-to-fmt.py `git log -n 1 "--pretty=%at" ../.`)

This won't work if ./configure is running in a source folder extracted from a release tarball.

• all common pages from osmo-gsm-manuals.git can be built as standalone test now (to see if there are any syntax errors etc. in the common pages, and to see if the common build scripts are working)
• made --enable-man work with out-of-tree builds
• unix-time-to-fmt.py: use "unknown" if we're not in a git directory
• both pdfs for msc get generated now, though there's content missing in the vty reference

• fixed missing pages in the vty-reference pdfs
• make checks from asciidoc Makefile work again
• update Makefile.*.inc text comments
• remove --publish from contrib/jenkins.sh, add a "publish" target to the common Makefile that gets included in each project

And I've thought about creating a "osmo-gsm-manuals-dev" Debian package as part of this issue. But there doesn't seem to be any benefit to that until we can build the manuals as man pages (#3386). So instead of blowing up the patchset further with a new "debian"-folder, let's do that as part of the man pages issue.

I'll clean up the code, then submit the patches for review on Monday :)

• https://gerrit.osmocom.org/#/q/topic:move-manuals+(status:open+OR+status:merged)

The manuals are only added to MSC so far. When this is reviewed, I can add them to the other project repositories as well.

One patch in the patchset ("build manuals from the project repositories") did too many things at once:
https://gerrit.osmocom.org/#/c/osmo-gsm-manuals/+/11725/

I have split it up in multiple commits, and refactored the code so all the existing, project specific manuals build with autotools with the current folder structure before we finally move them. That will make the moving a lot easier, because we already know that all of them build. And it makes it easier to see that the changes don't break anything.

New patchset coming in tomorrow, my current code (that still needs clean up before pushing to gerrit) is in the "osmith/move-manuals-to-project-repos" branches in osmo-msc.git and osmo-gsm-manuals.git.

The ~20 patches that change the buildsystem to autoconf/automake to get "make install" working have been merged (also with "make distcheck" support for consistency with the other Osmocom projects, so "make distcheck" will still work with ./configure --enable-manuals).

Jenkins tried to run ./jenkins.sh --publish and failed, I've patched the job so it doesn't do that anymore:
https://gerrit.osmocom.org/#/c/osmo-ci/+/11863/

neels: you proposed in here that we keep the git history when moving the files, bypassing gerrit. I've attached a script that does this. Please read it once, edit the commit message if you like, and if you think it is fine, would you like to run it and push the changes to master of each repository? On my laptop, this takes a bit more than four minutes to move all manuals with the history.

The checklist items added are the last missing pieces to finally finishing this up.

In argreement with Neels, I have executed the script and pushed the commits to an extra branch called osmith/move-manuals-with-history:

Left: dir name in osmo-gsm-manuals.git
Right: project git repo name

OsmoBSC => osmo-bsc
OsmoBTS => osmo-bts
OsmoGGSN => osmo-ggsn
OsmoGSMTester => osmo-gsm-tester
OsmoHLR => osmo-hlr
OsmoMGW => osmo-mgw
OsmoMSC => osmo-msc
OsmoNITB => openbsc
OsmoPCU => osmo-pcu
OsmoSGSN => osmo-sgsn
OsmoSIPConnector => osmo-sip-connector
OsmoSTP => libosmo-sccp
OsmoTRX => osmo-trx
OsmocomBB => osmocom-bb

These dirs have not been moved, as they are obsolete. They only contained the vty reference, which could be easily restored anyway.

OsmoMGCP: NOT MOVING
OsmoNAT: NOT MOVING

neels: thanks for applying the patch to these repositories:

OsmoBSC => osmo-bsc
OsmoBTS => osmo-bts
OsmoGGSN => osmo-ggsn
OsmoHLR => osmo-hlr
OsmoMGW => osmo-mgw
OsmoMSC => osmo-msc
OsmoPCU => osmo-pcu
OsmoSGSN => osmo-sgsn
OsmoSIPConnector => osmo-sip-connector
OsmoSTP => libosmo-sccp
OsmoTRX => osmo-trx

These following are still missing. They don't have a configure.ac in the top level, but the manuals can be built with cd doc/manuals; make as the commit message in their osmith/move-manuals-with-history branches says. Can you also merge those?

OsmoNITB => openbsc
OsmocomBB => osmocom-bb
OsmoGSMTester => osmo-gsm-tester

Patches for deletion of the moved manual dirs have been submitted:
https://gerrit.osmocom.org/#/q/topic:move-manuals+status:open

Regarding build of manuals in CI, it seems that the buildbots, on which the project related builds run, do not have all dependencies installed:

https://jenkins.osmocom.org/jenkins/job/gerrit-osmo-bsc/a1=default,a2=default,a3=default,a4=default,label=osmocom-gerrit-debian9/1877/console

Binary 'xsltproc' not found in path, please install libxslt.
configure: error: "missing dependencies!" 

EDIT: the buildbots have all dependencies, but some jobs are running in Docker containers, and these don't have the dependencies. I can patch this though. Here's some documentation of how it all fits together: https://osmocom.org/projects/osmocom-servers/wiki/Jenkins_build_verification_jobs

osmith wrote:

These following are still missing.

ah thanks, completed now

Getting closer to the finish line, new patches up for review.

https://gerrit.osmocom.org/#/q/topic:move-manuals+status:open

• osmo-mgw was built twice, with and without transcoding support, although it doesn't support that (leftover from old code)
• docker arguments were formatted differently in gerrit-verifications.yml and master-builds.yml, which made it harder to check both files side-by-side
• redundant environment variable specifications were confusing
• openbsc.git's manuals had wrong names (e.g. osmo-nat instead of osmo-bsc-nat)
• wrong distcheck variable was used in Makefile.am, so it could not be overruled in contrib/jenkins.sh

The patches for extending the contrib/jenkins.sh files went through several iterations (in which some of the points above came up), and are finally in master of the respective Osmocom git repositories. The Jenkins jobs are adjusted to pass environment variables for building and publishing manuals. I've updated the TODO list.

These should be the final three patches:
https://gerrit.osmocom.org/#/q/topic:move-manuals+status:open

Finished \o/