Project

General

Profile

Bug #3720

Jenkins can't upload to rita.osmocom.org

Added by osmith 6 days ago. Updated 32 minutes ago.

Status:
Resolved
Priority:
High
Assignee:
Target version:
-
Start date:
12/06/2018
Due date:
% Done:

100%

Spec Reference:

Description

On 22th of November, master-simtrace2 started failing because it can't upload to rita.osmocom.org anymore.

Warning: Permanently added the ECDSA host key for IP address '[144.76.43.76]:48' to the list of known hosts.
Permission denied (publickey,keyboard-interactive).
rsync: connection unexpectedly closed (0 bytes received so far) [sender]

https://jenkins.osmocom.org/jenkins/job/master-simtrace2/a1=default,a2=default,a3=default,label=osmocom-master-debian9/197/console

This is not related to the manuals refactoring that is going on in #3385, although uploading the manuals would use the same mechanism (so it is blocked by this issue).


Related issues

Related to Cellular Network Infrastructure - Bug #3725: Jenkins isn't using the credentials store for uploading to rita.osmocom.orgNew2018-12-12

Blocks Cellular Network Infrastructure - Feature #3385: Move project specific manuals from osmo-gsm-manuals to each respective git repositoryIn Progress2018-07-06

History

#1 Updated by laforge 6 days ago

  • Status changed from New to In Progress
  • Assignee set to osmith
  • % Done changed from 0 to 30

The debug log on the ssh server on ftp.osmocom.org looks like this:

Dec  6 12:41:57 ftp sshd[11359]: debug1: fd 4 clearing O_NONBLOCK
Dec  6 12:41:57 ftp sshd[11359]: debug1: Forked child 11363.
Dec  6 12:41:57 ftp sshd[11363]: debug1: rexec start in 4 out 4 newsock 4 pipe 6 sock 7
Dec  6 12:41:57 ftp sshd[11363]: debug1: inetd sockets after dupping: 3, 3
Dec  6 12:41:57 ftp sshd[11363]: debug1: res_init()
Dec  6 12:41:57 ftp sshd[11363]: debug1: Client protocol version 2.0; client software version OpenSSH_7.4p1 Debian-10+deb9u4
Dec  6 12:41:57 ftp sshd[11363]: debug1: match: OpenSSH_7.4p1 Debian-10+deb9u4 pat OpenSSH* compat 0x04000000
Dec  6 12:41:57 ftp sshd[11363]: debug1: Local version string SSH-2.0-OpenSSH_7.5 FreeBSD-20170903
Dec  6 12:41:57 ftp sshd[11363]: debug1: Enabling compatibility mode for protocol 2.0
Dec  6 12:41:57 ftp sshd[11363]: debug2: fd 3 setting O_NONBLOCK
Dec  6 12:41:57 ftp sshd[11363]: debug2: Network child is on pid 11364
Dec  6 12:41:57 ftp sshd[11363]: debug1: permanently_set_uid: 22/22 [preauth]
Dec  6 12:41:57 ftp sshd[11363]: debug1: list_hostkey_types: ssh-rsa,rsa-sha2-512,rsa-sha2-256,ecdsa-sha2-nistp256,ssh-ed25519 [preauth]
Dec  6 12:41:57 ftp sshd[11363]: debug1: SSH2_MSG_KEXINIT sent [preauth]
Dec  6 12:41:57 ftp sshd[11363]: debug1: SSH2_MSG_KEXINIT received [preauth]
Dec  6 12:41:57 ftp sshd[11363]: debug2: local server KEXINIT proposal [preauth]
Dec  6 12:41:57 ftp sshd[11363]: debug2: KEX algorithms: curve25519-sha256,curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,diffie-hellman-group14-sha256,diffie-hellman-group14-sha1 [preauth]
Dec  6 12:41:57 ftp sshd[11363]: debug2: host key algorithms: ssh-rsa,rsa-sha2-512,rsa-sha2-256,ecdsa-sha2-nistp256,ssh-ed25519 [preauth]
Dec  6 12:41:57 ftp sshd[11363]: debug2: ciphers ctos: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com,aes128-cbc,aes192-cbc,aes256-cbc [preauth]
Dec  6 12:41:57 ftp sshd[11363]: debug2: ciphers stoc: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com,aes128-cbc,aes192-cbc,aes256-cbc [preauth]
Dec  6 12:41:57 ftp sshd[11363]: debug2: MACs ctos: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1 [preauth]
Dec  6 12:41:57 ftp sshd[11363]: debug2: MACs stoc: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1 [preauth]
Dec  6 12:41:57 ftp sshd[11363]: debug2: compression ctos: none,zlib@openssh.com [preauth]
Dec  6 12:41:57 ftp sshd[11363]: debug2: compression stoc: none,zlib@openssh.com [preauth]
Dec  6 12:41:57 ftp sshd[11363]: debug2: languages ctos:  [preauth]
Dec  6 12:41:57 ftp sshd[11363]: debug2: languages stoc:  [preauth]
Dec  6 12:41:57 ftp sshd[11363]: debug2: first_kex_follows 0  [preauth]
Dec  6 12:41:57 ftp sshd[11363]: debug2: reserved 0  [preauth]
Dec  6 12:41:57 ftp sshd[11363]: debug2: peer client KEXINIT proposal [preauth]
Dec  6 12:41:57 ftp sshd[11363]: debug2: KEX algorithms: curve25519-sha256,curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha256,diffie-hellman-group14-sha1,ext-info-c [preauth]
Dec  6 12:41:57 ftp sshd[11363]: debug2: host key algorithms: ecdsa-sha2-nistp256-cert-v01@openssh.com,ecdsa-sha2-nistp384-cert-v01@openssh.com,ecdsa-sha2-nistp521-cert-v01@openssh.com,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,ssh-ed25519-cert-v01@openssh.com,ssh-rsa-cert-v01@openssh.com,ssh-ed25519,rsa-sha2-512,rsa-sha2-256,ssh-rsa [preauth]
Dec  6 12:41:57 ftp sshd[11363]: debug2: ciphers ctos: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com,aes128-cbc,aes192-cbc,aes256-cbc [preauth]
Dec  6 12:41:57 ftp sshd[11363]: debug2: ciphers stoc: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com,aes128-cbc,aes192-cbc,aes256-cbc [preauth]
Dec  6 12:41:57 ftp sshd[11363]: debug2: MACs ctos: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1 [preauth]
Dec  6 12:41:57 ftp sshd[11363]: debug2: MACs stoc: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1 [preauth]
Dec  6 12:41:57 ftp sshd[11363]: debug2: compression ctos: none,zlib@openssh.com,zlib [preauth]
Dec  6 12:41:57 ftp sshd[11363]: debug2: compression stoc: none,zlib@openssh.com,zlib [preauth]
Dec  6 12:41:57 ftp sshd[11363]: debug2: languages ctos:  [preauth]
Dec  6 12:41:57 ftp sshd[11363]: debug2: languages stoc:  [preauth]
Dec  6 12:41:57 ftp sshd[11363]: debug2: first_kex_follows 0  [preauth]
Dec  6 12:41:57 ftp sshd[11363]: debug2: reserved 0  [preauth]
Dec  6 12:41:57 ftp sshd[11363]: debug1: kex: algorithm: curve25519-sha256 [preauth]
Dec  6 12:41:57 ftp sshd[11363]: debug1: kex: host key algorithm: ecdsa-sha2-nistp256 [preauth]
Dec  6 12:41:57 ftp sshd[11363]: debug1: kex: client->server cipher: chacha20-poly1305@openssh.com MAC: <implicit> compression: none [preauth]
Dec  6 12:41:57 ftp sshd[11363]: debug1: kex: server->client cipher: chacha20-poly1305@openssh.com MAC: <implicit> compression: none [preauth]
Dec  6 12:41:57 ftp sshd[11363]: debug1: expecting SSH2_MSG_KEX_ECDH_INIT [preauth]
Dec  6 12:41:57 ftp sshd[11363]: debug2: monitor_read: 6 used once, disabling now
Dec  6 12:41:57 ftp sshd[11363]: debug2: set_newkeys: mode 1 [preauth]
Dec  6 12:41:57 ftp sshd[11363]: debug1: rekey after 134217728 blocks [preauth]
Dec  6 12:41:57 ftp sshd[11363]: debug1: SSH2_MSG_NEWKEYS sent [preauth]
Dec  6 12:41:57 ftp sshd[11363]: debug1: expecting SSH2_MSG_NEWKEYS [preauth]
Dec  6 12:41:57 ftp sshd[11363]: debug1: SSH2_MSG_NEWKEYS received [preauth]
Dec  6 12:41:57 ftp sshd[11363]: debug2: set_newkeys: mode 0 [preauth]
Dec  6 12:41:57 ftp sshd[11363]: debug1: rekey after 134217728 blocks [preauth]
Dec  6 12:41:57 ftp sshd[11363]: debug1: KEX done [preauth]
Dec  6 12:41:57 ftp sshd[11363]: debug1: userauth-request for user binaries service ssh-connection method none [preauth]
Dec  6 12:41:57 ftp sshd[11363]: debug1: attempt 0 failures 0 [preauth]
Dec  6 12:41:57 ftp sshd[11363]: debug2: parse_server_config: config reprocess config len 205
Dec  6 12:41:57 ftp sshd[11363]: debug2: monitor_read: 8 used once, disabling now
Dec  6 12:41:57 ftp sshd[11363]: debug2: input_userauth_request: setting up authctxt for binaries [preauth]
Dec  6 12:41:57 ftp sshd[11363]: debug1: PAM: initializing for "binaries" 
Dec  6 12:41:57 ftp sshd[11363]: debug1: PAM: setting PAM_RHOST to "build-2.osmocom.org" 
Dec  6 12:41:57 ftp sshd[11363]: debug2: monitor_read: 100 used once, disabling now
Dec  6 12:41:57 ftp sshd[11363]: debug2: input_userauth_request: try method none [preauth]
Dec  6 12:41:57 ftp sshd[11363]: debug1: userauth-request for user binaries service ssh-connection method publickey [preauth]
Dec  6 12:41:57 ftp sshd[11363]: debug1: attempt 1 failures 0 [preauth]
Dec  6 12:41:57 ftp sshd[11363]: debug2: input_userauth_request: try method publickey [preauth]
Dec  6 12:41:57 ftp sshd[11363]: debug1: userauth_pubkey: test whether pkalg/pkblob are acceptable for RSA SHA256:vji0ppcVgc9SCguHuN8gg3CLqgiWadYkgTlSlqBreO8 [preauth]
Dec  6 12:41:57 ftp sshd[11363]: debug2: monitor_read: 4 used once, disabling now
Dec  6 12:41:57 ftp sshd[11363]: debug1: trying public key file /home/binaries/.ssh/authorized_keys
Dec  6 12:41:57 ftp sshd[11363]: debug1: fd 4 clearing O_NONBLOCK
Dec  6 12:41:57 ftp sshd[11363]: debug2: key not found
Dec  6 12:41:57 ftp sshd[11363]: debug1: trying public key file /home/binaries/.ssh/authorized_keys2
Dec  6 12:41:57 ftp sshd[11363]: debug1: Could not open authorized keys '/home/binaries/.ssh/authorized_keys2': No such file or directory
Dec  6 12:41:57 ftp sshd[11363]: debug2: userauth_pubkey: authenticated 0 pkalg rsa-sha2-512 [preauth]
Dec  6 12:41:57 ftp sshd[11363]: debug1: userauth-request for user binaries service ssh-connection method keyboard-interactive [preauth]
Dec  6 12:41:57 ftp sshd[11363]: debug1: attempt 2 failures 1 [preauth]
Dec  6 12:41:57 ftp sshd[11363]: debug2: input_userauth_request: try method keyboard-interactive [preauth]
Dec  6 12:41:57 ftp sshd[11363]: debug1: keyboard-interactive devs  [preauth]
Dec  6 12:41:57 ftp sshd[11363]: debug1: auth2_challenge: user=binaries devs= [preauth]
Dec  6 12:41:57 ftp sshd[11363]: debug1: kbdint_alloc: devices 'pam' [preauth]
Dec  6 12:41:57 ftp sshd[11363]: debug2: auth2_challenge_start: devices pam [preauth]
Dec  6 12:41:57 ftp sshd[11363]: debug2: kbdint_next_device: devices <empty> [preauth]
Dec  6 12:41:57 ftp sshd[11363]: debug1: auth2_challenge_start: trying authentication method 'pam' [preauth]
Dec  6 12:41:57 ftp sshd[11363]: debug2: monitor_read: 104 used once, disabling now

So basically the key was simply rejected as the key on client and server don't agree.

I don't think anything has changed on the ftp jail on rita.osmocom.org.

It contains the followng keys:

ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDZc2rcIY29MvVQh/9pZmikrL5jlTomk0J9hIogDqSNaUanpguU9CcquHcLvbVAMh0HVjHexm/BQCCnxwEfBnQ490PAEMjZGACMffcjahr63/FqThHIk906jPXdLkWDc70E/PEJdAxuGGGxhZA//+is/YvDR6j3lBy/7CiJdx2xwlsCGQZ6KMHpepGdUGoBuUvGLNRtGu+lGpBC/NdQl1FAn527Z+cqqfW9Eq8a2H0MRSQV8lwFZYR7/B2BXDYiQHxN7eOYXIYVezEG870Tpabfvfigj5Tpt3rQGNLW/sAN1+DQJkhc4sPFfzfK6Y/OQFuq8YvM/p5LtAEpByIgtmxJ builder@vagrant-ubuntu-wily-64
ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDWBcxVO1mKLVmRI2aHJn4SB8IlN4ZGRtqcD72/Xun4D6oU4ywB65AzEEJwWYfRFRjj+Uy4JZLDYPcracuToqK9go1FPM25bsk/TmS821pja+vpeCMVfLq5jnsnwgzKrWKhRCpnviKPTeyLfw7wkF1fk5xp133j9z0IElSjiZRQxbQKDQMAMZ313CQ6XouDGmeMoV5g0cQ6ahLweFIHLkpIkwZhiAQnUgHmSwh+SrzY+sIqKucWdJrB7vLktqbt5gjwfBpBVnsXQY9RCoh8T4w2LXLhXRqOwSsaMW6Z9Pji6pYbppvbXH4Qu/EY337seAGv7CSOzT5i1QW10Mhgy5tP osmocom-build@deb8build
ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDjFrV8TP4IBcFb+S72EMM06VPiujBxHnGe+w4mgeWWi7ZgjekFgkRIZfVjTm9R78gK1Ybj+MiMwXVD/cmj9RQkbHT1dS/Sj7btEvIxKgz8mEtIgA0DC/UQUE0NRJqz0mjrbITc9sKz8vTM/Mvvvs1NeHeT0N6K53wnYqKBQduvqYhM/dotNxlkZdDZUiCOLv5beYUpNz8u8AHJtf717l82KtjaXm5DbncrAXRT3rVTvS0QTELapPzk9XNx8TB3ALxmaB2WUkhvHtBS4geOCRel/kwbglbqCNVzXYf7YTBij3YbASrRQTt9nTmsgArg4VY0bkqDDv2qIMB8pG4viS11 kevredon@dennou
ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCmaAZWIdZZjFZHq7Xm5OCUz/Cm46haad+pm2iZ2uSQCIXubjN8HGz1XujBMH1DHhSDK+L0JOBEtxHaY4JR7F177V5R8o50VNwmtTrTaNBPS8WPyVBDuHjjDVSZA+d6QT3Y+3X/FhfoFKmGc2a8MZRmE8yDGW484y7MTZ4XBHqy5vWnnywxWCZm7kVqNccM4oQ4tPWlMTxk6+tgWC4wvk3axsUVUm0ak/WKuZLoX+jBWlsWXR4uwaOEqblpNMv98CxjLgpjEBTvZxgrgY35eAm11S9lmvxFSjLIZA1V/2huAA/9MSNvI4dX4ZQPHfLCsMqFjhkIY0hvFK2uX8hlxHhOR6GkOq3n3MdcKCMI+JAISCP6g++DdQO5wzfOLeSdpTETNpIxeFkjQLS+HnuAcgjRK6O7czI+xYfCiBcaoslsTIYi33Pg7gPuDjwAyVdKqQPVdYPluR8W5+FT1lFtf1jwIB3jgt+v9q+r6d4LfkOCAU17yxG6b40P+FqK9O7X+iUkva/lbghdAw7DimczDIuEMpVVVL3/WhmCIMrfJPG/RtMdoczcAw4q4FxBretZR1Ke4AomUvUklFwfMtLqijxBJN2Owpy7jtqggs+v6cUX54fKh7/SeVrgyIgeUtgJp6lqaxlMao8SoR59YoqLqZueYOp9QxYujA70AGoMMwxsEw== laforge@nataraja

However, the individual debian9 build slaves [now] contain each separate RSA keys for the osmocom-build user:

ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCyUsYxX5Ro53cq8JK+kpd6J6IsugsnF4F4W40tX3E4OfTklOewCAhrSgeZd1tKk96XUoK2lRrlZEzU4e/mN0vS4R7SkfqOjmNlzzIVRRi6vTXD4iseNcSSM76SVENRKYsMrXfBQAzcYP5IiaSiMxOz1dDiCKRV4FErlWvv5C60ekGPCtZudCqZZQ7gsSO+PjwW3h/53U5mLte1RpS1sgGmMz3ZzUD6m/ZtChYCrwIPxYT0DRFeq2HlZHCMJvQCPlNp9B5WoeKHPP+FtnTi3WMzd4GpsTNhb85p5QnF2GhGIH54bxjjajK0G1/O9XsTNkugZwCEmyBpL8FtAVAAMla7w30qBr5+2bjc1sb9l6pHTohcH9QLs0y3rJJilscdHeiWCejDidRxNe1o1BzJnsQNHkfU108Fz0+OWF9U/Iro45sd77r/5DL2MYGnBpvC3GC2K1Les1fFKsfZqLGLLhLgv5z/jKfi76eyi5yj6mYuV+NfYI6y/pPIEDw64Cff/1Lcl8jVZ5hNsAADEuVxEydUpoqnXk/nF7/Yonuqsu/N32lZING2OIbz4m4qqtzyPnLH6yyCDAxRw5b2rLT6jcrWJOE7Rni5yyhVwPfIwGrrB5yH5qQRFhrzQTVbiQZCWSokH0kvhMlu4GPD7aXU5BrDDBYipdzLxN4qPvBXigFSsQ== ansible-generated on build2-deb9build-ansible
ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDgWYeuVQHL8537ZhlOxWlF+40ieNN4NlSii1+igCBdOhkSq4nY+/XeKxoB7rs7DlY0DPSe10x+LBNZccXCGSupl+4ugjoH0x7bNyBJHomELaiUftc8W1Oz8TR7lmlesxAdqwXvxiSCVAsygXulW4isVBaHFWhz1Q9fqAwT/cQBMHrXb4z1+qf4iwKEe1jYBLZmrRrvaTj+dup9oFxztpakcivGmqlxVhKRMurkiwbrtIZj4ByfzOITXIts2szbBqursQlQCkFiwye/o/kQbRxdBgWTMyxkdJMvRohLwiyNq5+auFpKzebkkLAnaH8014G5lujlUX4AB9n5pTzv6CatPdsv1gQdnM6rkqjs/xiXAPtO5SfTMM9d/LcYpFyNlAZwTXcJAXHAHxU6A0xSQC1s+H258PL/9icSregcD0QyQ6Ymi5UyCgW+mGtDx4H4OBR8nMO+B3OKN1kGmP6zOyUI9407yRF7+bgJkFjJsfIZq0/HUpAPTGD0/f30mg3hlBPOpe4fl071oH9hWg49GkJCZSr6YKNNYRKrbKR7TVfGyhR+GHyKxH665cF49b8lXD2s0hHKH2gOdGEOdC+gR6EUOo3bnSFt1ET1spUq+k2Kth+wbZ/pCXFGD5WXw8P/qoQy8UlNVk64woixURMLNd7mkpuEX6jOXZto2pP60Lmd8w== ansible-generated on deb9build-ansible
ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDJc7O0Oht1T6qfVDOE6EB+P6aRsalOHrBevuNlFM8ZzVE/PcOCYBJMHVm60tNezv6a2SSSAVMYElVtlDlXRjl4+reAsrrP0ppSALvUyxqTpq3o/wjpvBsxL+lvZtktS+gfFXjCMOPQ42eTuUcuLmYFdZkSV77yAux1JVYgOJ1Pn/W64g9gA3ER8oDef96VWON7pu8nHOeTPbOui3L1zmGzC4JSYjXv5Op3j0Df/THKdfZPrmCcKXp36JwKWBqbWQ/aS30nKzFdMajZ8kVq4PTLzskEVuitiTWecuAxZd2mfMFU/zfuj5mN3cB/TCbi+I3FGuod8NwvOCnRYRWcmEaZTPnWNQKlmUpZ4BIMyV0DR6NBRWD0HYYHzDTGKbiH6FDWBmsETrjfaM4movEqBHXUC3MgkjXM5EFDdOL4In08W5ZsJd++AmmQU95FpylLeAZT+HoQMuZlrEHzJrxgJ5w2ReBTQW2doFISXpdcaOKTPkd3kbFTCQi4uDLT+OG4Pij2EuSuN6GQ7mpT41Gow81L4RyvQEdbf0Bw9XFegPTVi4RqTwfktjDkSFz08rgFm0mLWl1R15BmHvPXgWolT+dDV7G0DA2Z3pT/HVhWz1xXt3DiWzi6b9YIEceCeZZRpNGBJdk5v8FKw6f7Au4bKiVeouWlyp88XrveUbgJ3PqIaQ== ansible-generated on osmo-build-debian9

So the only causes I could imagine are:
  • somebody re-generated/overwrote the ~/.ssh/id_rsa on the build slaves, or
  • those jobs uploading to ftp.osmocom.org were previously ran only on one specific build slave, which must have [had] the key listed on the server. the key name "osmocom-build@deb8build" strongly hints to that. Maybe somebody moved those jobs from debian8 to debian9 slaves?

As an interim measure, I've added the three debian9 jails' individual RSA public keys as stated above to the ftp server authorized_keys. I've verified this manually.

The proper solution is to use the jenkins server credentials store, which will hold the private key and provision it to the client via ssh-agent. That way the client can upload to the server, and no per-slave configuration is required on the ftp server.

#2 Updated by laforge 6 days ago

  • Blocks Feature #3385: Move project specific manuals from osmo-gsm-manuals to each respective git repository added

#3 Updated by osmith 5 days ago

With this workaround, master-simtrace2 is not failing anymore. I ran it again today, just to be sure.

However, the workaround does not work for publishing the PDFs from the Osmocom projects, they still get a permission denied. I've tested this with osmo-mgw (inside Docker) and osmo-hlr (does not run inside Docker).

https://jenkins.osmocom.org/jenkins/view/master/job/master-osmo-mgw/1884/
https://jenkins.osmocom.org/jenkins/view/master/job/master-osmo-hlr/1681/

Comparing both jobs, I've noticed that:

laforge: can you verify if the keys are also added for the "docs" user, and add them there if they are missing?

I would suggest we finish up #3385 with this workaround in place (so the manuals get published properly again).

Afterwards look into using SSH agent, building all projects inside Docker (not only some, as you have suggested on the mailing list), and making sure the publish part runs outside of Docker after building.

#4 Updated by osmith 32 minutes ago

  • Status changed from In Progress to Resolved
  • % Done changed from 30 to 100

laforge adjusted the config, and now the manuals can be published.

Created #3725 for using the credentials store.

#5 Updated by osmith 31 minutes ago

  • Related to Bug #3725: Jenkins isn't using the credentials store for uploading to rita.osmocom.org added

Also available in: Atom PDF

Add picture from clipboard (Maximum size: 48.8 MB)