Cellular Network Infrastructure

h1. Contributing using Gerrit

{{>toc}}

At [[OpenBSC:OsmoDevCon2016]] we discussed problems with our past contribution / patch submission process using mails on the mailing list as well as patchwork.  The result was that we wanted to give Gerrit a try for some time and see if it helps us to have a better process.  We were (and still are) quite happy with it, and over time most Osmocom projects have migrated to using gerrit.

Gerrit is a review tool that integrates nicely with git and ssh. You can find general information about Gerrit at https://www.gerritcodereview.com/

The advantages of Gerrit are:

* patch submission status is automatically tracked, also with several revisions for a patch set.

* patches are build-tested (and possibly even further tested) by jenkins before they are applied

* patch submissions not via git send-email but direcly from git

h2. Osmocom Subprojects using Gerrit

The majority of Osmocom sub-projects have chosen to use Gerrit for patch review.  In order to check if a given program uses Gerrit, please check the auto-generated list at https://gerrit.osmocom.org/#/admin/projects/

If the project is listed there, then it uses Gerrit.   If the project is not listed there, please send patches by e-mail to the respective project [[Mailing_Lists]] instead.

h2. Configuring Gerrit/Account

You will need to sign up at https://gerrit.osmocom.org/login/. If you have an Osmocom Redmine account you can use https://osmocom.org/openid as OpenID provider.

* first sign in on https://osmocom.org. Do this before logging into gerrit (the redmine login process loses the gerrit login data and you'd have to do the same thing twice if not logged in on osmocom.org already).

* go to https://gerrit.osmocom.org and click the "Sign in" link.

* click the "Sign in with Osmocom":https://gerrit.osmocom.org/login/%23%2Fq%2Fstatus%3Aopen?id=https://osmocom.org/openid link (can be bookmarked). -- This is the same as entering https://osmocom.org/openid as OpenID provider and hitting the "Sign in" button.

*careful:* enter 'https' to ensure that your openid credentials are passed on encrypted.

*pitfall:* if you're logged in on 'projects.osmocom.org' (including the 'projects.' part), you should also use the openid provider: https://projects.osmocom.org/openid; the 'projects.' part may be omitted, what's important is that redmine login and OpenID URLs match. Also, decide for one of those URLs once, because when picking a different OpenID URL next time, you will create a new user instead of logging in as yourself.

*note:* gerrit will create a distinct user for each openid URL you pass. If you logged in successfully but your user seems to have lost permissions, you may have created an evil twin user: contact us on the mailing list so we can fix it in the user database.

If you have no Osmocom redmine account, you can simply create one online at the "Register" link in the upper right corner.

Even without an existing or new redmine account, you should also be able to use any other OpenID provider to authenticate against gerrit (untested).

After the initial sign-up you will need to:

* Pick a username (cannot be changed)

* Add your public ssh key(s)

* Add email addresses you intend to use as author/comitter

If you would like to push private branches to the Gerrit repository, you also need to be added to the "known users" group.

Please send a short requesting email to openbsc@lists.osmocom.org.

h2. Setting up Gerrit for commits and pushing

*Note:* it is easiest to work with gerrit when gerrit is the only remote in your git clone.

When you clone from gitea.osmocom.org and add the gerrit remote, git will have two remotes,

so when you first checkout a branch you have to supply the remote explicitly (cumbersome).

The gerrit repositories and gitea.osmocom.org are constantly synced, so it is sufficient

to clone from gerrit only.

h3. Simplest: new clone

* Create a new clone from gerrit

<pre><code class="shell">

git clone ssh://$USERNAME@gerrit.osmocom.org:29418/$PROJECT.git

</code></pre>

* Fetch the commit hook that adds Change-Id to each commit to uniquely identify a commit

<pre><code class="shell">

# Option a) using scp

scp -P 29418 $USERNAME@gerrit.osmocom.org:hooks/commit-msg $PROJECT/.git/hooks/

# Option b) using curl

curl -o $PROJECT/.git/hooks/commit-msg https://gerrit.osmocom.org/tools/hooks/commit-msg

chmod +x $PROJECT/.git/hooks/commit-msg

</code></pre>

h3. SSH config

In '~/.ssh/config', add these lines:

<pre>

Host go

Hostname gerrit.osmocom.org

Port 29418

User $USERNAME

</pre>

('go' means gerrit.osmocom, replace with your favorite shortcut name,

replace '$USERNAME' with your user name as used on the gerrit website)

Then you can shorten above commands to

<pre>

git clone ssh://go/$PROJECT.git

cd $PROJECT

scp go:hooks/commit-msg .git/hooks/

</pre>

h3. Committer must match

Your email address on gerrit and the email address git places in your

commits must match, or you will get rejected with an error message like

"invalid committer". You can add email addresses on the gerrit web UI.

h3. Add gerrit to an existing clone

* Add the remote to be able to fetch and push to gerrit

<pre><code class="shell">

USERNAME=gerrit_user_name

PROJECT=$(basename $PWD)

git remote add gerrit ssh://$USERNAME@gerrit.osmocom.org:29418/$PROJECT.git

</code></pre>

* Fetch the commit hook that adds Change-Id to each commit to uniquely identify a commit

<pre><code class="shell">

# Option a) using scp

scp -P 29418 $USERNAME@gerrit.osmocom.org:hooks/commit-msg .git/hooks/

# Option b) using curl

curl -o .git/hooks/commit-msg https://gerrit.osmocom.org/tools/hooks/commit-msg

chmod +x .git/hooks/commit-msg

</code></pre>

h2. Push for review

Prerequisites:

* your user on gerrit has an SSH public key

* your patch is committed in your local clone

* the commit log message has a Change-Id (see 'commit-msg' hook above, and 'Tips and Tricks' below to add a Change-Id to a commit that lacks one.)

<pre>

git push $REMOTE $GITHASH:refs/for/$BRANCH%topic=$TOPIC

</pre>

$REMOTE: from above instructions, that's either 'origin' (cloned from gerrit) or 'gerrit' (if you added a second remote).

$GITHASH: the committed patch to push, typically you're on your branch and simply push 'HEAD'.

$BRANCH: you will typically intend a patch to go to 'master'.

%topic=$TOPIC: an optional name you may choose.

Pushing as WIP _with a topic_ does *NOT* work like that, you need to do:

<pre>

git push $REMOTE $GITHASH:refs/for/$BRANCH%wip -o topic=$TOPIC

</pre>

For example, check out the revision or branch that you want to submit for review,

i.e. the one where your patch or several patches are committed on top of the current master, then:

If you cloned directly from gerrit:

<pre>

git push origin HEAD:refs/for/master

</pre>

If you added 'gerrit' as a second remote to an existing clone:

<pre>

git push gerrit HEAD:refs/for/master

</pre>

You can optionally add a topic name with

<pre>

git push origin HEAD:refs/for/master%topic=my_topic

</pre>

h2. Voting Rules for merging a patch to master

High-level goal: Two people _other than the patch author/owner_ should have given positive review ("Review +1") _and_ the patch must pass automatic build/lint validation ("Verified: +1").

Please follow these voting rules:

h3. Code Review ("CR")

* Please review patches by others.

* Do not vote for your own patches (exceptions below).

* Before merging, each patch should receive at least two reviews that approve merging.

When voting, please follow this social contract (written from the point of a _reviewer_):

* When you approve, vote CR +1.

* If there already is someone else's CR +1, you *as a reviewer, not as the author/owner* may also choose to vote CR +2.

* If the patch owner sees two or more CR +1, the patch owner may apply to self a CR +2.

* Once there is at least one CR +2 and one CR +1 vote, a patch may be merged ("Submit" button), except:

** If there are two -1 votes, you should not merge, instead clarify the reason and try to fix it.

** If there is a single -1 vote, you may still merge the patch, if you are sure that the opinion of the -1 vote does not carry.

* Give the benefit of the doubt to the -1 vote, do not lightheartedly overrule.

* If there is a CR -2 vote, the patch will likely never pass review, it marks a fundamental flaw.

* Merging a patch ("Submit" button) may be done by a reviewer or by the patch owner,

  But if there is any negative vote, rather leave merging up to the patch owner.

* If you remove a +1 vote, try to make sure that there are no other CR +2 votes left alone

  (to prevent accidental "Submit including parents"). If needed, ping other reviewer / admin on IRC.

  But try to vote +1 only when you're sure, hence this situation should be rare.

* To prevent merging of your own patch before some issue is resolved, consider marking it Work In Progress.

h3. Exceptions for Trivial / Urgent Patches

A patch may receive a direct +2 when:

* it is very trivial, like a typo fix in a comment or log string, so that it is not worth wasting review time on.

* it reverts an earlier change that broke the master builds.

In these cases, the patch submitter may decide to +2 to self, after careful consideration. This should be rare.

h3. Verification ("V")

* For most projects, jenkins takes care of voting V +1 automatically.

* If you have actually tried out a patch and verified that it works, you may vote V +1.

* A patch owner may vote V +1 to self in a project that has no Jenkins verification job.

h3. Rationale

Gerrit allows merging a patch as soon as a CR +2 vote and a V +1 vote are in.

For quite some time, we had CR +2 permissions for only very few gatekeeper reviewers.

The result was that non-gatekeepers' votes seemed to not matter.

To encourage more peer review that actually has an effect, we would like to sum up +1 votes.

We have tried to apply a summing of votes in Gerrit with automatic enforcing

( https://gerrit-review.googlesource.com/Documentation/prolog-cookbook.html#_example_13_1_1_2_code_review )

but this had numerous quirks, particularly the issues summary shows wildly mismatching voting status.

The solution is to agree on a social contract: everyone gets +2 permissions,

but you shall only use it when it makes sense. Thanks!

h2. Manage private branches

Use a sub-directory with your name to group your own branches, please.

*Note* that you must be a member of the "known users" group, see above.

To share / backup local branches to gitea.osmocom.org, without starting a code review process on gerrit.osmocom.org, just push them as usual to gerrit.

* Note that the git repository must be cloned from the gerrit SSH URL -- all pushing goes to gerrit ("pushurl" as described below also works).

* All private branches are automatically synced to gitea.osmocom.org in a matter of minutes.

* Private branches do not kick off patch sets for review, they are just branches. To kick off review for all patches on your branch, you'd use a 'refs/for/master' URL, as shown in the example below.

*Pitfall*: always include both your name and a topic in a branch name, right from the start. If you first create a branch named "fred", creating branches named "fred/topic" later will cause conflicts in the .git admin tree, because first "fred" was a file, and then becomes a directory. Even if you fix it in your local repository, *all* other git repositories, including those on the build servers, may need manual action to become usable again. One solution is to push a deletion of "fred" and then "git fetch --purge" everywhere else.

The typical transcript for "Fred" developing feature "Kazoo" looks like this:

<pre>

# First, set up ~/.ssh/config so that 'go' points at gerrit.osmocom.org.

# Then:

git clone ssh://go/sandbox

cd sandbox

# create local branch

git checkout -b fred/kazoo

$EDITOR

git add kazoo.c

git commit -m "implement kazoo"

# just invoke 'git push' and git tells you how to create the branch once-off:

git push

|fatal: The current branch kazoo has no upstream branch.

|To push the current branch and set the remote as upstream, use

|

|    git push --set-upstream origin fred/kazoo

git push --set-upstream origin fred/kazoo

# Now the branch exists on gerrit, very soon will also exist on gitea.osmocom.org

# Further tweaks

$EDITOR

git add kazoo.h

git commit --amend

# You are free to force-push private branches

git push -f

# origin/master has changed? Rebase onto the latest.

git fetch

git rebase -i origin/master

# resolve conflicts...

git push -f

# Feature is ready, submit the entire branch to Gerrit code review

git push origin HEAD:refs/for/master/kazoo

</pre>

h2. List changesets in gerrit

<pre>

git ls-remote gerrit changes/*

</pre>

h1. Tips and Tricks

h2. A commit lacks a Change-Id

once you added the commit hook as above, just re-edit the commit log message, e.g. with

<pre>

git commit --amend

</pre>

or by

<pre>

git rebase -i

</pre>

and in the upcoming editor replacing 'pick' with 'r' in front of the commit to edit.

No need to change the commit log if you don't want to, just exit the editor and the commit hook will add a Change-Id.

h2. Ignore WIP patches

Using following operators in "search" field of web UI allows to ignore Work-in-progress changes:

<pre>

status:open AND -is:wip

</pre>

h2. Fetch fast from gitea.osmocom.org, push to gerrit

Gerrit has moved to a faster host, so this should no longer be necessary. Anyway...

Adding a second remote forces you to explicitly pass the remote on the command line ("origin").

It is possible to have only one remote for convenience, with different push and pull URLs:

<pre>

git remote set-url origin https://gitea.osmocom.org/$ORG/$PROJECT

git remote set-url --push origin ssh://$USERNAME@gerrit.osmocom.org:29418/$PROJECT

</pre>

With above .ssh config you can also use the shorter ssh:// URL:

<pre>

git remote set-url --push origin ssh://go/$PROJECT

</pre>

The resulting .git/config in libosmocore would look something like:

<pre>

[remote "origin"]

        url = https://gitea.osmocom.org/osmocom/libosmocore

        pushurl = ssh://go/libosmocore

        fetch = +refs/heads/*:refs/remotes/origin/*

</pre>

Now you're fetching from gitea.osmocom.org, which is lightning fast, while pushing patches will still go to gerrit as usual.

h2. Throw-away branch

If you need to adjust and re-submit patches, it may be handy to create a throw-away branch ("R D" in magit-gerrit in emacs for example),

make your changes/amendments and then send patch(es) back to gerrit while removing temporary branch automatically with "git review -f".

h2. Fetch a patch from gerrit

This script (I called it @P@) makes fetching a patch set from gerrit a breeze:

<pre>

#!/bin/sh

# fetch gerrit patch into new branch named like the patch number.

#

# Usage: go to a git clone and pass a patch number:

#

#   cd openbsc

#   P 973

# or

#   P 973/2

#

# Will create new local branches '973_4' (if 4 is the latest patch set)

# or '973_2', respectively.

patch="$1"

if [ -z "$patch" ]; then

  echo "Usage: P 1234[/5]"

  exit 1

fi

if [ -z "$(echo "$patch" | grep '/')" ]; then

  patch="/$patch/"

fi

if [ -z "$(echo "$patch" | grep '^/')" ]; then

  patch="/$patch"

fi

remote="$(git remote get-url --push origin)"

last_set="$(git ls-remote "$remote" "changes/*" | grep "$patch" | sed 's#.*/\([^/]*\)$#\1 &#' | sort -n | tail -n 1)"

if [ -z "$last_set" ]; then

  echo "Not found: $patch"

  exit 1

fi

change_name="$(echo "$last_set" | sed 's/.*\(refs.*\)/\1/')"

branch_name="$(echo "$change_name" | sed 's#refs/changes/../\([0-9]*\)/\([0-9]*\)#\1_\2#')"

set -x

git fetch "$remote" "$change_name"

git co -b "$branch_name" FETCH_HEAD

</pre>

h2. Re-submit a Branch with Amended Commits

On a feature branch, one typically has numerous commits that depend on their preceding commits.

Often, some of the branch commits need to be amended for fixes. You can re-submit changes to

patches on your branch by pushing in the same way that you first submitted the branch.

Note: if you modify the Change-Ids in the commit logs, your push would open entirely new

review entries and you would have to abandon your previous submission. Comments on the first

submission are "lost" and you cannot diff between patch sets.

h2. Re-submit Previously Abandoned Changes

You have to edit the Change-Ids, on a branch that would be every single commit log message.

<pre>

cd openbsc

git co my-branch

git rebase -i master

# replace all 'pick' with 'r' (or 'reword'), exit your editor

# git presents each commit log message for editing

</pre>

h2. 502 Bad Gateway

When getting a "Bad Gateway" error message upon trying to login on gerrit, you probably just need to restart your web browser. The reason is not clear.

h2. scp fails with "subsystem request failed on channel 0"

Most likely, you're using a very recent OpenSSH release (9.0 - check https://www.openssh.com/txt/release-9.0 43 for more info), which changes the scp program to use the SFTP protocol under the hood.

<pre>

$ scp -P 29418 gerrit.osmocom.org:hooks/commit-msg .git/hooks/

subsystem request failed on channel 0

scp: Connection closed

</pre>

To work around the problem on the client side, use the new @-O@ (that is an uppercase letter "o") switch when invoking scp, which will cause it to fall back to the legacy behavior.

<pre>

$ scp -O -P 29418 gerrit.osmocom.org:hooks/commit-msg .git/hooks/

commit-msg                                              100% 2174     9.5KB/s   00:00

</pre>

h2. Commit hook: Always put Change-Id at the bottom of the log message

The commit-msg hook places a Change-Id tag in the footer, often above other tags like 'Depends:' or 'Related:'. Since the Change-Id is an implementation detail for Gerrit, I personally prefer it always placed right at the bottom. This simple edit changes the commit-msg hook to add Change-Id at the bottom unconditionally:

<pre>

cd $PROJECT

sed -i 's/if (unprinted /if (0 \&\& unprinted /' .git/hooks/commit-msg

</pre>

The goal is to disable the condition in line 163 with an 'if (0...':

<pre>

                        if (0 && unprinted && match(tolower(footer[line]), changeIdAfter) != 1) {

                                unprinted = 0

                                print "Change-Id: I'"$id"'"

                        }

</pre>

Then the Change-Id will be placed by line 170 instead.

h1. Reasons for Particular Configuration

h2. Rebase if necessary

There are different merge strategies that Gerrit performs to accept patches.

Each project can be configured to a specific merge strategy, but unfortunately you can't

decide on a strategy per patch submission.

It seems that the "Merge if Necessary" strategy is best supported, but it creates non-linear

history with numerous merge commits that are usually not at all necessary.

Instead, the "Cherry Pick" strategy puts each patch onto current master's HEAD to create

linear history. However, this will cause merge failures as soon as one patch depends on

another submitted patch, as typical for a feature branch submission.

So we prefer the "Rebase if Necessary" strategy, which always tries to apply your patches to

the current master HEAD, in sequence with the previous patches on the same branch.

h2. Private Branches: Create a new change for every commit...

Say you have an extensive feature in development, and you want to keep it on the

upstream git repository to a) keep it safe and b) collaborate with other devs on it.

So, of course, you have regularly pushed to refs/heads/yoyodyne/feature.

Since you have the gerrit commit hook installed, your feature branch already has

Change-Id tags in all commit log messages.

Now your feature is complete and you would like to submit it to master.

Alas, Gerrit refuses to accept your patch submission for master, because it

knows the Change-Ids are also on a different branch.

Gerrit by default enforces that a Change-Id must be unique across all branches,

so that each submission for review is separate for each branch. Instead, we

want to handle Change-Ids per-branch, so that you can have the same change

submitted to different branches, as separate patch submissions, without having

to cosmetically adjust the Change-Id.

Solution: set the option 

_Create a new change for every commit not in the target branch_ to _TRUE_

h2. Allow content merges

By default, gerrit compares patches only by the files' paths. If two paths are the same,

it immediately shows them as conflicts (path conflicts).

In software development, a conflict usually means an actual content conflict, so if the

edits are in two entirely separate places in the file, we don't consider this a conflict.

By setting _Allow content merges_ to _TRUE_ in the git project config, we tell Gerrit to

perform text merges of the submitted patches and only complain about actual content

conflicts, in the usual software engineering sense.

h1. Admin

h2. Adding a new repository

1. create the repository in the Gerrit UI, inherit from "All-Projects"

2. a) project does not exist on gitea.osmocom.org:

* create an empty git repository using gitea on https://gitea.osmocom.org/

2. b) project exists on https://gitea.osmocom.org:

* explicitly add gerrit as a second remote and push all branches/tags from gitea.osmocom.org to gerrit.

3. configure a jenkins build testing job for this project (see gerrit-verifications.yml in osmo-ci.git/jobs)

git replication to gerrit.osmocom.org is enabled automatically, nothing to be done here.  In case of doubt, try

@ssh -p 29418 gerrit.osmocom.org replication start --all --wait@

h2. Deleting a repository

@ssh -p 29418 gerrit.osmocom.org delete-project delete --yes-really-delete NAME@

h2. Adding users to groups

Normally, the gerrit UI auto-completes a user name in the edit field. It has happened

though that an existing user is not auto-completed, as if it didn't exist. In that case,

find out the user ID (seven digit number like 1000123) and just enter that.

The user ID can be found on the user's "Settings" page, or in the database (s.b.).

h2. Querying the database directly

If your user has permission to access the database, you can place SQL queries using the

'gerrit gsql' commands over ssh:

<pre>

ssh go "gerrit gsql -c \"show tables\""

ssh go "gerrit gsql -c \"select full_name,account_id from accounts\""

</pre>

(see ~/.ssh/config above for the 'go' shortcut)

This seems to be the MySQL dialect.

The "...\"...\"" quoting allows including single-quotes in the SQL statements.

h2. Fix evil twin users

If differing openid URLs have lead to evil twin users shadowing the same email address just without the permissions, we end up with duplicate users showing up in the UI, e.g. when you want to add somebody as a reviewer.

In past gerrit vesions based on ReviewDB, one could use SQL statements to manipulate the database. However, gerrit has migrated to NoteDB, and hence this access is no longer possible.

You can find information on how to manipulate users using the REST API at

* https://charm.cs.illinois.edu/gerrit/Documentation/rest-api-accounts.html#account-endpoints

* https://charm.cs.illinois.edu/gerrit/Documentation/rest-api.html

You can get a list of identities for one given e-mail address (e.g. @pmaier@sysmocom.de@) using

 curl 'https://gerrit.osmocom.org/accounts/?q=pmaier@sysmocom.de'

which would give you a result like this:

<pre>

[

  {

    "_account_id": 1000028

  },

  {

    "_account_id": 1000140

  }

]

</pre>

And subsequetly check which of the identities you want to keep or remove:

 curl --user http_user:http_pass  'https://gerrit.osmocom.org/a/accounts/1000028/external.ids'

curl --user http_user:http_pass  'https://gerrit.osmocom.org/a/accounts/1000140/external.ids'

The "intended" ones typically have https://osmocom.org/openid URLs, while the unintended/dupliacte ones have http (not https) or use www.osmocom.org or projects.osmocom.org instead.

Once you have found the duplicate (in this case 100140), you can deactivate the duplicate using 

 curl --user http_user:http_pass -X DELETE 'https://gerrit.osmocom.org/a/accounts/1000140/active'