Project

General

Profile

Actions

Bug #4862

closed

replicate download.opensuse.org content somewhere on *.osmocom.org

Added by laforge about 1 year ago. Updated 29 days ago.

Status:
Resolved
Priority:
Normal
Assignee:
Category:
-
Target version:
-
Start date:
11/18/2020
Due date:
% Done:

100%

Spec Reference:

Description

as we can see right now. {download,mirrors,build}.opensuse.org are all down.

This means we can not build any of our container images, nor can any users install pre-built osmocom patches.

We should at least have an alternative mirror on osmocom.org.


Checklist

  • move to dedicated uid
  • export via rsync
  • export via https
  • reference it from wiki so it can be found

Related issues

Related to Osmocom.org Servers - Feature #5083: migrate jenkins master to machine with lots of HDD storageResolvedlaforge03/20/2021

Actions
Actions #1

Updated by laforge about 1 year ago

https://ftp.uni-stuttgart.de/opensuse/repositories/network:/osmocom:/ seems to be a current mirror, in case anyone needs it.

Actions #2

Updated by pespin about 1 year ago

See patch:
https://gerrit.osmocom.org/c/docker-playground/+/21239 common: Introduce OSMOCOM_REPO_MIRROR param

With it, one can for instance:
cd ttcn3-pcu-test
OSMOCOM_REPO_MIRROR="http://ftp.uni-stuttgart.de/opensuse" ./jenkins.sh

Actions #3

Updated by laforge 8 months ago

  • Related to Feature #5083: migrate jenkins master to machine with lots of HDD storage added
Actions #4

Updated by laforge 8 months ago

now that we have host3.osmocom.org with 4TB RAID-1 storage, we should be able to host both amirror of the OBS builds, as well as an archive of them.

Actions #5

Updated by laforge 8 months ago

Something like rsync://62.146.92.210/buildservice-repos/network:/osmocom: works to rsync the packages. The size is quite small:

2.4M    ./pcsc
293M    ./latest
3.1M    ./mingw:
8.0K    ./tdm:
4.0K    ./next
8.0K    ./nitb-split:
175M    ./nightly
Actions #6

Updated by laforge 8 months ago

  • Status changed from Stalled to In Progress
  • % Done changed from 0 to 30

I'm testing the following script now on host3:

#!/bin/bash
set -e -x

# base directory on the local side
BASE_DIR=/home/laforge/osmocom-obs-mirror/
# sync remote where to find the osmocom packages
REMOTE=rsync://ftp.gwdg.de/pub/opensuse/repositories/network:/osmocom:

cd "$BASE_DIR" 

RSYNC_ARGS="-av --delete" 
DATE=`date +%Y%m%d-%H%M%S`

# create output directory
DIR="$BASE_DIR/$DATE" 
mkdir -p "$DIR" 

PREVIOUS="$BASE_DIR/previous" 
if [ -d "$PREVIOUS" ]; then
        RSYNC_ARGS+=" --link-dest=$PREVIOUS" 
fi

# finally, perform rsync
rsync $RSYNC_ARGS "$REMOTE"/ "$DIR"/

# update 'previous' for the next run
rm "$PREVIOUS" 
ln -sf "$DATE" "$PREVIOUS" 
Actions #7

Updated by laforge 7 months ago

  • Checklist item move to dedicated uid added
  • Checklist item export via rsync added
  • Checklist item export via https added
  • Checklist item reference it from wiki so it can be found added

the script seems to be working fine. de-duplication works as expected. Currently using 19GB for 10 days of history (twice daily).

Need to make this available publicly via at least https and rsync and document it for people to find

Actions #8

Updated by laforge 7 months ago

  • Checklist item move to dedicated uid set to Done
  • Checklist item export via rsync set to Done
  • % Done changed from 30 to 60
$ rsync rsync://host3.osmocom.org/obs-mirror/
drwxr-xr-x          4,096 2021/04/22 09:54:05 .
lrwxrwxrwx             15 2021/04/22 00:23:20 previous
drwxr-xr-x          4,096 2020/11/19 22:48:52 20210412-111633
drwxr-xr-x          4,096 2020/11/19 22:48:52 20210412-111812
drwxr-xr-x          4,096 2020/11/19 22:48:52 20210412-111848
drwxr-xr-x          4,096 2020/11/19 22:48:52 20210412-111930
drwxr-xr-x          4,096 2020/11/19 22:48:52 20210412-112218
drwxr-xr-x          4,096 2020/11/19 22:48:52 20210412-122301
drwxr-xr-x          4,096 2020/11/19 22:48:52 20210413-002301
drwxr-xr-x          4,096 2020/11/19 22:48:52 20210413-122301
drwxr-xr-x          4,096 2020/11/19 22:48:52 20210414-002301
drwxr-xr-x          4,096 2020/11/19 22:48:52 20210414-122301
drwxr-xr-x          4,096 2020/11/19 22:48:52 20210415-002301
drwxr-xr-x          4,096 2020/11/19 22:48:52 20210415-122301
drwxr-xr-x          4,096 2020/11/19 22:48:52 20210416-002301
drwxr-xr-x          4,096 2020/11/19 22:48:52 20210416-122301
drwxr-xr-x          4,096 2020/11/19 22:48:52 20210417-002301
drwxr-xr-x          4,096 2020/11/19 22:48:52 20210417-122301
drwxr-xr-x          4,096 2020/11/19 22:48:52 20210418-002301
drwxr-xr-x          4,096 2020/11/19 22:48:52 20210418-122301
drwxr-xr-x          4,096 2020/11/19 22:48:52 20210419-002301
drwxr-xr-x          4,096 2020/11/19 22:48:52 20210419-122301
drwxr-xr-x          4,096 2020/11/19 22:48:52 20210420-002301
drwxr-xr-x          4,096 2020/11/19 22:48:52 20210420-122301
drwxr-xr-x          4,096 2020/11/19 22:48:52 20210421-002302
drwxr-xr-x          4,096 2020/11/19 22:48:52 20210421-122301
drwxr-xr-x          4,096 2020/11/19 22:48:52 20210422-002301
Actions #9

Updated by laforge 7 months ago

  • Checklist item export via https set to Done
Actions #10

Updated by laforge 7 months ago

  • Checklist item reference it from wiki so it can be found set to Done
  • Status changed from In Progress to Resolved
  • % Done changed from 60 to 100

Linked from wiki at Latest_Builds and Nightly_Builds

Actions #11

Updated by osmith 6 months ago

Script updated:
  • "previous" -> ".previous" (so it does not show up in dir listing)
  • "rm -f" instead of "rm", so it doesn't stop if the previous symlink does not exist and never creates it (as we just ran into this)

obs-mirror.sh

Actions #12

Updated by osmith 6 months ago

Updated:
  • only archive directories mentioned in obs-mirror-include.txt
  • add "|| true" after rsync command so it doesn't stop there if a dir from the list was not found
  • add test-include-list.sh

obs-mirror.sh

obs-mirror-include.txt

test-include-list.sh

Actions #13

Updated by osmith 6 months ago

As discussed, I've added a new cronjob to delete nightly archives older than one month.

rm-old-nightly-archives.sh

output

Actions #14

Updated by osmith 29 days ago

as discussed, updated to download from rsync.opensuse.org in order to have rpm debug packages available too.

obs-mirror.sh

Actions

Also available in: Atom PDF

Add picture from clipboard (Maximum size: 48.8 MB)