Bug #2746
closedprod env: osmo-ggsn unable to get IPv6 link-local address
100%
Description
gprs tests are currently failing when trying to create an ipv6 pdp context.
The issue is actually in osmo-ggsn returning an error when sgsn sends a Pdp Activate with IPv4v6. When the modem asks for an ipv4, it still uses a IPv4v6 (see #2713) but when the error is returned it re-tries with a IPv4-only EUA and then it works. However for IPv6 it doesn't and it returns a failure through the dbus interface.
Now the interesting part for this ticket: How osmo-ggsn fails.
When receiving a request for an ipv4v6 ctx, osmo-ggsn tries to get one IP of each type from the pools in the APN. Before looking in the pool, it checks if the APN actually supports this kind of address / has this type of pool. When checking for the IPv6 pool, it fails printing the following + sending a packet back with an error:
20171213174828668 DGGSN <0002> ggsn.c:635 PDP(901700000015252:6): APN doesn't support requested EUA / AF type
That happens because the following path triggers:
/* does this APN actually have an IPv6 pool? */ if (!apn_supports_ipv6(apn)) goto err_wrong_af;
which means the ipv6 pool was not configured successfully at startup. We can actually see it in log file:
20171213174741477 DGGSN <0002> ggsn.c:201 APN(internet): Starting 20171213174741477 DGGSN <0002> ggsn.c:204 APN(internet): Opening TUN device tun4 20171213174741478 DGGSN <0002> ggsn.c:209 APN(internet): Opened TUN device tun4 20171213174741478 DGGSN <0002> ggsn.c:220 APN(internet): Setting tun IP address 176.16.222.1/24 20171213174741480 DGGSN <0002> ggsn.c:283 APN(internet): Creating IPv4 pool 176.16.222.0/24 20171213174741481 DGGSN <0002> ggsn.c:185 APN(internet): Blacklist tun IP 176.16.222.1/24 20171213174741481 DIP <0000> ippool.c:295 addr blacklisted from pool: 176.16.222.1 20171213174741481 DGGSN <0002> ggsn.c:314 APN(internet): Successfully started 20171213174741482 DGGSN <0002> ggsn.c:201 APN(inet6): Starting 20171213174741482 DGGSN <0002> ggsn.c:204 APN(inet6): Opening TUN device tun6 20171213174741482 DGGSN <0002> ggsn.c:209 APN(inet6): Opened TUN device tun6 20171213174741482 DGGSN <0002> ggsn.c:232 APN(inet6): Setting tun IPv6 address fde4:8dba:82e1:2000::/56 20171213174741484 DGGSN <0002> ggsn.c:252 APN(inet6): Cannot obtain IPv6 link-local address of interface: Success 20171213174741484 DGGSN <0002> ggsn.c:115 APN(inet6): Stopping 20171213174741484 DGGSN <0002> ggsn.c:128 APN(inet6): Closing TUN device tun6 20171213174741503 DGGSN <0002> ggsn.c:201 APN(inet46): Starting 20171213174741503 DGGSN <0002> ggsn.c:204 APN(inet46): Opening TUN device tun46 20171213174741504 DGGSN <0002> ggsn.c:209 APN(inet46): Opened TUN device tun46 20171213174741504 DGGSN <0002> ggsn.c:220 APN(inet46): Setting tun IP address 176.16.46.0/24 20171213174741505 DGGSN <0002> ggsn.c:232 APN(inet46): Setting tun IPv6 address fde4:8dba:82e1:2000::/56 20171213174741506 DGGSN <0002> ggsn.c:252 APN(inet46): Cannot obtain IPv6 link-local address of interface: Success 20171213174741506 DGGSN <0002> ggsn.c:115 APN(inet46): Stopping 20171213174741506 DGGSN <0002> ggsn.c:128 APN(inet46): Closing TUN device tun46 20171213174741535 DGGSN <0002> ggsn.c:812 GGSN(ggsn0): Starting GGSN
As you can see, it fails to configure ipv6 pools because it fails to get the IPv6 link-local address from the tun iface:
20171213174741506 DGGSN <0002> ggsn.c:252 APN(inet46): Cannot obtain IPv6 link-local address of interface: Success
Later on it fails with different errno:
20171213174741549 DGGSN <0002> ggsn.c:252 APN(inet6): Cannot obtain IPv6 link-local address of interface: No such file or directory
The only path in tun/netdev_ip_local_get which can return an error and issues a syscall is:
if (getifaddrs(&ifaddr) == -1) { return -1; }
This function only receives a pointer and has no context associated with our code. It actually works fine in R&D env (debian9 in R&D vs debian 8 in Prod), which means there's a bug in libc or kernel in debian8.
debian8 (Prod):
- Debian GLIBC 2.19-18+deb8u10
- #1 SMP Debian 3.16.43-2+deb8u2 (2017-06-26)
debian9 (R&D):
- Debian GLIBC 2.24-11+deb9u1
- kernel: 1 SMP Debian 4.9.30-2+deb9u3 (2017-08-06)
"man getifaddrs":
RETURN VALUE On success, getifaddrs() returns zero; on error, -1 is returned, and errno is set appropriately. ERRORS getifaddrs() may fail and set errno for any of the errors specified for socket(2), bind(2), getsockname(2), recvmsg(2), sendto(2), malloc(3), or realloc(3). VERSIONS The getifaddrs() function first appeared in glibc 2.3, but before glibc 2.3.3, the implementation supported only IPv4 addresses; IPv6 support was added in glibc 2.3.3. Support of address families other than IPv4 is available only on kernels that support netlink.
So regarding glibc version it should support ipv6 addresses (it doesn't mean there aren't bugs in there though), and "kernel that support" netlink it's a big difficult to match to a specific version.
In any case, it seems like another good reason to upgrade production environment to debian 9.
Files
Updated by pespin over 6 years ago
Upgrading latest packages with apt-get (staying in debian8) + rebooting didn't help solving the issue.
Updated by pespin over 6 years ago
I did a quick test by running the following program while the test was running and it never failed:
#include <stdlib.h> #include <stdio.h> #include <sys/types.h> #include <ifaddrs.h> int main(int argc, char** argv) { struct ifaddrs *ifaddr; if (getifaddrs(&ifaddr) == -1) { fpritnf(stderr, "fail\n"); return 1; } else fprintf(stderr, "success\n"); return 0; }
Closer look showed me that actually the check is "tun_ip_local_get(apn->tun.tun, &ipv6_tun_linklocal_ip, 1, IP_TYPE_IPv6_LINK) < 1" which means it checks not only for error, but asserts that the IP was found. Which means the getifaddrs is working fine but the address is not found. I run "ifconfig" and "ip l" while the test was running, and I only saw tun4 appearing. However, according to osmo-ggsn logs we see no error appearing.
Updated by pespin over 6 years ago
After stracing osmo-ggsn, there seems to be something weird in the output of while setting ipv6 SIOCSIFADDR tun addr (tun_setaddr6):
write(2, "20171213202024691 DGGSN <0002> g"..., 6820171213202024691 DGGSN <0002> ggsn.c:201 APN(inet6): Starting ) = 68 write(2, "20171213202024693 DGGSN <0002> g"..., 8320171213202024693 DGGSN <0002> ggsn.c:204 APN(inet6): Opening TUN device tun6 ) = 83 open("/dev/net/tun", O_RDWR) = 6 ioctl(6, TUNSETIFF, 0x7ffe4d18a690) = 0 ioctl(6, TUNSETNOCSUM, 0x1) = 0 write(2, "20171213202024696 DGGSN <0002> g"..., 8220171213202024696 DGGSN <0002> ggsn.c:209 APN(inet6): Opened TUN device tun6 ) = 82 fcntl(6, F_GETFL) = 0x8002 (flags O_RDWR|O_LARGEFILE) fcntl(6, F_SETFL, O_RDWR|O_NONBLOCK|O_LARGEFILE) = 0 fcntl(6, F_GETFD) = 0 fcntl(6, F_SETFD, FD_CLOEXEC) = 0 write(2, "20171213202024697 DGGSN <0002> g"..., 10920171213202024697 DGGSN <0002> ggsn.c:232 APN(inet6): Setting tun IPv6 address fde4:8dba:82e1:2000::/56 ) = 109 access("/proc/net", R_OK) = 0 access("/proc/net/unix", R_OK) = 0 socket(PF_LOCAL, SOCK_DGRAM|SOCK_CLOEXEC, 0) = 7 ioctl(7, SIOCGIFINDEX, {ifr_name="tun6", ifr_index=42}) = 0 close(7) = 0 socket(PF_INET6, SOCK_DGRAM, IPPROTO_IP) = 7 ioctl(7, SIOCSIFADDR, {ifr_name="<FD>䍺<82><E1> ", ifr_addr={0x38 /* AF_??? */, "\0\0*\0\0\0\1\0\0\0\0\0\0\0"}}) = 0 close(7) = 0 socket(PF_INET, SOCK_DGRAM, IPPROTO_IP) = 7 ioctl(7, SIOCSIFFLAGS, {ifr_name="tun6", ifr_flags=IFF_UP|IFF_RUNNING}) = 0 close(7) = 0 socket(PF_NETLINK, SOCK_RAW, NETLINK_ROUTE) = 7 bind(7, {sa_family=AF_NETLINK, pid=0, groups=00000000}, 12) = 0 getsockname(7, {sa_family=AF_NETLINK, pid=4567, groups=00000000}, [12]) = 0 sendto(7, "\24\0\0\0\22\0\1\3x}1Z\0\0\0\0\0\0\0\0", 20, 0, {sa_family=AF_NETLINK, pid=0, groups=00000000}, 12) = 20 recvmsg(7, {msg_name(12)={sa_family=AF_NETLINK, pid=0, groups=00000000}, msg_iov(1)=[{"l\4\0\0\20\0\2\0x}1Z\327\21\0\0\0\0\4\3\1\0\0\0I\0\1\0\0\0\0\0"..., 4096}], msg_controllen=0, msg_flags=0}, 0) = 3412 recvmsg(7, {msg_name(12)={sa_family=AF_NETLINK, pid=0, groups=00000000}, msg_iov(1)=[{"t\4\0\0\20\0\2\0x}1Z\327\21\0\0\0\0\1\0\4\0\0\0\3\20\0\0\0\0\0\0"..., 4096}], msg_controllen=0, msg_flags=0}, 0) = 3412 recvmsg(7, {msg_name(12)={sa_family=AF_NETLINK, pid=0, groups=00000000}, msg_iov(1)=[{"t\4\0\0\20\0\2\0x}1Z\327\21\0\0\0\0\1\0\7\0\0\0\2\20\0\0\0\0\0\0"..., 4096}], msg_controllen=0, msg_flags=0}, 0) = 3412 recvmsg(7, {msg_name(12)={sa_family=AF_NETLINK, pid=0, groups=00000000}, msg_iov(1)=[{"p\4\0\0\20\0\2\0x}1Z\327\21\0\0\0\0\1\0\n\0\0\0\2\20\0\0\0\0\0\0"..., 4096}], msg_controllen=0, msg_flags=0}, 0) = 3392 recvmsg(7, {msg_name(12)={sa_family=AF_NETLINK, pid=0, groups=00000000}, msg_iov(1)=[{"\24\0\0\0\3\0\2\0x}1Z\327\21\0\0\0\0\0\0", 4096}], msg_controllen=0, msg_flags=0}, 0) = 20 sendto(7, "\24\0\0\0\26\0\1\3y}1Z\0\0\0\0\0\0\0\0", 20, 0, {sa_family=AF_NETLINK, pid=0, groups=00000000}, 12) = 20 recvmsg(7, {msg_name(12)={sa_family=AF_NETLINK, pid=0, groups=00000000}, msg_iov(1)=[{"L\0\0\0\24\0\2\0y}1Z\327\21\0\0\2\10\200\376\1\0\0\0\10\0\1\0\177\0\0\1"..., 4096}], msg_controllen=0, msg_flags=0}, 0) = 1468 recvmsg(7, {msg_name(12)={sa_family=AF_NETLINK, pid=0, groups=00000000}, msg_iov(1)=[{"H\0\0\0\24\0\2\0y}1Z\327\21\0\0\n\200\200\376\1\0\0\0\24\0\1\0\0\0\0\0"..., 4096}], msg_controllen=0, msg_flags=0}, 0) = 288 recvmsg(7, {msg_name(12)={sa_family=AF_NETLINK, pid=0, groups=00000000}, msg_iov(1)=[{"\24\0\0\0\3\0\2\0y}1Z\327\21\0\0\0\0\0\0", 4096}], msg_controllen=0, msg_flags=0}, 0) = 20 close(7) = 0 write(2, "20171213202024703 DGGSN <0002> g"..., 11920171213202024703 DGGSN <0002> ggsn.c:252 APN(inet6): Cannot obtain IPv6 link-local address of interface: Success
However all return codes look fine (matches what we see in the logs).
I attach full strace.
EDIT: I forgot to say, the strace is taken as root, and I verified that indeed only tun4 was created, so it's not a permissions issue.
Updated by pespin over 6 years ago
Here's the same output in R&D (which creates the tuns correctly):
write(2, "20171213204857407 DGGSN <0002> g"..., 6820171213204857407 DGGSN <0002> ggsn.c:201 APN(inet6): Starting ESC[0;m) = 68 write(2, "20171213204857408 DGGSN <0002> g"..., 8320171213204857408 DGGSN <0002> ggsn.c:204 APN(inet6): Opening TUN device tun6 ESC[0;m) = 83 open("/dev/net/tun", O_RDWR) = 6 ioctl(6, TUNSETIFF, 0x7fff7c1ba790) = 0 ioctl(6, TUNSETNOCSUM, 0x1) = 0 write(2, "20171213204857411 DGGSN <0002> g"..., 8220171213204857411 DGGSN <0002> ggsn.c:209 APN(inet6): Opened TUN device tun6 ESC[0;m) = 82 fcntl(6, F_GETFL) = 0x8002 (flags O_RDWR|O_LARGEFILE) fcntl(6, F_SETFL, O_RDWR|O_NONBLOCK|O_LARGEFILE) = 0 fcntl(6, F_GETFD) = 0 fcntl(6, F_SETFD, FD_CLOEXEC) = 0 write(2, "20171213204857412 DGGSN <0002> g"..., 10920171213204857412 DGGSN <0002> ggsn.c:232 APN(inet6): Setting tun IPv6 address fde4:8dba:82e1:2000::/56 ESC[0;m) = 109 access("/proc/net", R_OK) = 0 access("/proc/net/unix", R_OK) = 0 socket(AF_UNIX, SOCK_DGRAM|SOCK_CLOEXEC, 0) = 7 ioctl(7, SIOCGIFINDEX, {ifr_name="tun6", }) = 0 close(7) = 0 socket(AF_INET6, SOCK_DGRAM, IPPROTO_IP) = 7 ioctl(7, SIOCSIFADDR, {ifr_name="\375\344\215\272\202\341 ", ifr_addr={sa_family=0x38 /* AF_??? */, sa_data="\0\0\272\1\0\0\1\0\0\0\0\0\0\0"}}) = 0 close(7) = 0 socket(AF_INET, SOCK_DGRAM, IPPROTO_IP) = 7 ioctl(7, SIOCSIFFLAGS, {ifr_name="tun6", ifr_flags=IFF_UP|IFF_RUNNING}) = 0 close(7) = 0 socket(AF_NETLINK, SOCK_RAW, NETLINK_ROUTE) = 7 bind(7, {sa_family=AF_NETLINK, nl_pid=0, nl_groups=00000000}, 12) = 0 getsockname(7, {sa_family=AF_NETLINK, nl_pid=25384, nl_groups=00000000}, [12]) = 0 sendto(7, {{len=20, type=0x12 /* NLMSG_??? */, flags=NLM_F_REQUEST|0x300, seq=1513194537, pid=0}, "\0\0\0\0"}, 20, 0, {sa_family=AF_NETLINK, nl_pid=0, nl_groups=00000000}, 12) = 20 recvmsg(7, {msg_name={sa_family=AF_NETLINK, nl_pid=0, nl_groups=00000000}, msg_namelen=12, msg_iov=[{iov_base=[{{len=1232, type=0x10 /* NLMSG_??? */, flags=NLM_F_MULTI, seq=1513194537, pid=25384}, "\0\0\4\3\1\0\0\0I\0\1\0\0\0\0\0\7\0\3\0lo\0\0\10\0\r\0\1\0\0\0"...}, {{len=1240, type=0x10 /* NLMSG_??? */, flags=NLM_F_MULTI, seq=1513194537, pid=25384}, "\0\0\1\0\2\0\0\0C\20\1\0\0\0\0\0\t\0\3\0eth0\0\0\0\0\10\0\r\0"...}, {{len=1240, type=0x10 /* NLMSG_??? */, flags=NLM_F_MULTI, seq=1513194537, pid=25384}, "\0\0\1\0\3\0\0\0C\20\1\0\0\0\0\0\t\0\3\0eth1\0\0\0\0\10\0\r\0"...}, {{len=3352834104, type=0x7fdd /* NLMSG_??? */, flags=0, seq=3374275523, pid=32733}, "\334\5\0\0\0\0\0\00080\330\307\335\177\0\0\260z?\311\335\177\0\0\310\244\33|\377\177\0\0"...}], iov_len=4096}], msg_iovlen=1, msg_controllen=0, msg_flags=0}, 0) = 3712 recvmsg(7, {msg_name={sa_family=AF_NETLINK, nl_pid=0, nl_groups=00000000}, msg_namelen=12, msg_iov=[{iov_base=[{{len=1240, type=0x10 /* NLMSG_??? */, flags=NLM_F_MULTI, seq=1513194537, pid=25384}, "\0\0\1\0\4\0\0\0\3\20\0\0\0\0\0\0\t\0\3\0eth2\0\0\0\0\10\0\r\0"...}, {{len=1236, type=0x10 /* NLMSG_??? */, flags=NLM_F_MULTI, seq=1513194537, pid=25384}, "\0\0\1\0\5\0\0\0\2\20\0\0\0\0\0\0\n\0\3\0wlan0\0\0\0\10\0\r\0"...}, {{len=1244, type=0x10 /* NLMSG_??? */, flags=NLM_F_MULTI, seq=1513194537, pid=25384}, "\0\0\1\0Q\0\0\0\2\20\0\0\0\0\0\0\22\0\3\0wwp0s18f2u1u"...}, {{len=3374275523, type=0x7fdd /* NLMSG_??? */, flags=0, seq=1500, pid=0}, "80\330\307\335\177\0\0\260z?\311\335\177\0\0\310\244\33|\377\177\0\0\304\244\33|\377\177\0\0"...}], iov_len=4096}], msg_iovlen=1, msg_controllen=0, msg_flags=0}, 0) = 3720 recvmsg(7, {msg_name={sa_family=AF_NETLINK, nl_pid=0, nl_groups=00000000}, msg_namelen=12, msg_iov=[{iov_base=[{{len=1248, type=0x10 /* NLMSG_??? */, flags=NLM_F_MULTI, seq=1513194537, pid=25384}, "\0\0\1\0R\0\0\0\2\20\0\0\0\0\0\0\24\0\3\0wwp0s18f2u1u"...}, {{len=1220, type=0x10 /* NLMSG_??? */, flags=NLM_F_MULTI, seq=1513194537, pid=25384}, "\0\0\376\377S\0\0\0\220\20\0\0\0\0\0\0\24\0\3\0wwp0s18f2u1u"...}, {{len=1236, type=0x10 /* NLMSG_??? */, flags=NLM_F_MULTI, seq=1513194537, pid=25384}, "\0\0\1\0T\0\0\0\2\20\0\0\0\0\0\0\n\0\3\0wwan1\0\0\0\10\0\r\0"...}, {{len=0, type=0 /* NLMSG_??? */, flags=0, seq=524293, pid=0}}], iov_len=4096}], msg_iovlen=1, msg_controllen=0, msg_flags=0}, 0) = 3704 recvmsg(7, {msg_name={sa_family=AF_NETLINK, nl_pid=0, nl_groups=00000000}, msg_namelen=12, msg_iov=[{iov_base=[{{len=1224, type=0x10 /* NLMSG_??? */, flags=NLM_F_MULTI, seq=1513194537, pid=25384}, "\0\0\376\377U\0\0\0\220\20\0\0\0\0\0\0\24\0\3\0wwp0s18f2u1u"...}, {{len=1236, type=0x10 /* NLMSG_??? */, flags=NLM_F_MULTI, seq=1513194537, pid=25384}, "\0\0\1\0V\0\0\0\2\20\0\0\0\0\0\0\n\0\3\0wwan2\0\0\0\10\0\r\0"...}, {{len=1228, type=0x10 /* NLMSG_??? */, flags=NLM_F_MULTI, seq=1513194537, pid=25384}, "\0\0\376\377\271\1\0\0Q\0\1\0\0\0\0\0\t\0\3\0tun4\0\0\0\0\10\0\r\0"...}, {{len=0, type=0 /* NLMSG_??? */, flags=0, seq=524293, pid=0}}], iov_len=4096}], msg_iovlen=1, msg_controllen=0, msg_flags=0}, 0) = 3688 recvmsg(7, {msg_name={sa_family=AF_NETLINK, nl_pid=0, nl_groups=00000000}, msg_namelen=12, msg_iov=[{iov_base=[{{len=1228, type=0x10 /* NLMSG_??? */, flags=NLM_F_MULTI, seq=1513194537, pid=25384}, "\0\0\376\377\272\1\0\0Q\0\1\0\0\0\0\0\t\0\3\0tun6\0\0\0\0\10\0\r\0"...}, {{len=131088, type=0x8429 /* NLMSG_??? */, flags=NLM_F_REQUEST|NLM_F_DUMP_INTR|NLM_F_DUMP_FILTERED|0x5a00, seq=25384, pid=65536}, "V\0\0\0\2\20\0\0\0\0\0\0\n\0\3\0wwan2\0\0\0\10\0\r\0\350\3\0\0"...}], iov_len=4096}], msg_iovlen=1, msg_controllen=0, msg_flags=0}, 0) = 1228 recvmsg(7, {msg_name={sa_family=AF_NETLINK, nl_pid=0, nl_groups=00000000}, msg_namelen=12, msg_iov=[{iov_base=[{{len=20, type=NLMSG_DONE, flags=NLM_F_MULTI, seq=1513194537, pid=25384}, "\0\0\0\0"}, {{len=442, type=0x51 /* NLMSG_??? */, flags=NLM_F_REQUEST, seq=0, pid=196617}, "tun6\0\0\0\0\10\0\r\0\364\1\0\0\5\0\20\0\0\0\0\0\5\0\21\0\0\0\0\0"...}, {{len=0, type=0 /* NLMSG_??? */, flags=0, seq=0, pid=0}}], iov_len=4096}], msg_iovlen=1, msg_controllen=0, msg_flags=0}, 0) = 20 sendto(7, {{len=20, type=0x16 /* NLMSG_??? */, flags=NLM_F_REQUEST|0x300, seq=1513194538, pid=0}, "\0\0\0\0"}, 20, 0, {sa_family=AF_NETLINK, nl_pid=0, nl_groups=00000000}, 12) = 20 recvmsg(7, {msg_name={sa_family=AF_NETLINK, nl_pid=0, nl_groups=00000000}, msg_namelen=12, msg_iov=[{iov_base=[{{len=76, type=0x14 /* NLMSG_??? */, flags=NLM_F_MULTI, seq=1513194538, pid=25384}, "\2\10\200\376\1\0\0\0\10\0\1\0\177\0\0\1\10\0\2\0\177\0\0\1\7\0\3\0lo\0\0"...}, {{len=88, type=0x14 /* NLMSG_??? */, flags=NLM_F_MULTI, seq=1513194538, pid=25384}, "\2\30\200\0\2\0\0\0\10\0\1\0\n\t\31e\10\0\2\0\n\t\31e\10\0\4\0\n\t\31\377"...}, {{len=88, type=0x14 /* NLMSG_??? */, flags=NLM_F_MULTI, seq=1513194538, pid=25384}, "\2\30\200\0\3\0\0\0\10\0\1\0\n**\1\10\0\2\0\n**\1\10\0\4\0\n**\377"...}, {{len=88, type=0x14 /* NLMSG_??? */, flags=NLM_F_MULTI, seq=1513194538, pid=25384}, "\2\30\201\0\3\0\0\0\10\0\1\0\n**\2\10\0\2\0\n**\2\10\0\4\0\n**\377"...}, {{len=88, type=0x14 /* NLMSG_??? */, flags=NLM_F_MULTI, seq=1513194538, pid=25384}, "\2\30\201\0\3\0\0\0\10\0\1\0\n**\3\10\0\2\0\n**\3\10\0\4\0\n**\377"...}, {{len=88, type=0x14 /* NLMSG_??? */, flags=NLM_F_MULTI, seq=1513194538, pid=25384}, "\2\30\201\0\3\0\0\0\10\0\1\0\n**\4\10\0\2\0\n**\4\10\0\4\0\n**\377"...}, {{len=88, type=0x14 /* NLMSG_??? */, flags=NLM_F_MULTI, seq=1513194538, pid=25384}, "\2\30\201\0\3\0\0\0\10\0\1\0\n**\5\10\0\2\0\n**\5\10\0\4\0\n**\377"...}, {{len=88, type=0x14 /* NLMSG_??? */, flags=NLM_F_MULTI, seq=1513194538, pid=25384}, "\2\30\201\0\3\0\0\0\10\0\1\0\n**\6\10\0\2\0\n**\6\10\0\4\0\n**\377"...}, {{len=88, type=0x14 /* NLMSG_??? */, flags=NLM_F_MULTI, seq=1513194538, pid=25384}, "\2\30\201\0\3\0\0\0\10\0\1\0\n**\7\10\0\2\0\n**\7\10\0\4\0\n**\377"...}, {{len=88, type=0x14 /* NLMSG_??? */, flags=NLM_F_MULTI, seq=1513194538, pid=25384}, "\2\30\201\0\3\0\0\0\10\0\1\0\n**\10\10\0\2\0\n**\10\10\0\4\0\n**\377"...}, {{len=88, type=0x14 /* NLMSG_??? */, flags=NLM_F_MULTI, seq=1513194538, pid=25384}, "\2\30\201\0\3\0\0\0\10\0\1\0\n**\t\10\0\2\0\n**\t\10\0\4\0\n**\377"...}, {{len=88, type=0x14 /* NLMSG_??? */, flags=NLM_F_MULTI, seq=1513194538, pid=25384}, "\2\30\201\0\3\0\0\0\10\0\1\0\n**2\10\0\2\0\n**2\10\0\4\0\n**\377"...}, {{len=88, type=0x14 /* NLMSG_??? */, flags=NLM_F_MULTI, seq=1513194538, pid=25384}, "\2\30\201\0\3\0\0\0\10\0\1\0\n**3\10\0\2\0\n**3\10\0\4\0\n**\377"...}, {{len=88, type=0x14 /* NLMSG_??? */, flags=NLM_F_MULTI, seq=1513194538, pid=25384}, "\2\30\200\0\4\0\0\0\10\0\1\0\n@\1\1\10\0\2\0\n@\1\1\10\0\4\0\n@\1\377"...}, {{len=80, type=0x14 /* NLMSG_??? */, flags=NLM_F_MULTI, seq=1513194538, pid=25384}, "\2\30\200\0\4\0\0\0\10\0\1\0\300\250\1\310\10\0\2\0\300\250\1\310\t\0\3\0eth2"...}, {{len=80, type=0x14 /* NLMSG_??? */, flags=NLM_F_MULTI, seq=1513194538, pid=25384}, "\2\30\200\0\271\1\0\0\10\0\1\0\260\20\336\1\10\0\2\0\260\20\336\1\t\0\3\0tun4"...}, {{len=0, type=0x24 /* NLMSG_??? */, flags=NLM_F_MULTI|NLM_F_ACK|NLM_F_ECHO, seq=0, pid=0}}], iov_len=4096}], msg_iovlen=1, msg_controllen=0, msg_flags=0}, 0) = 1380 recvmsg(7, {msg_name={sa_family=AF_NETLINK, nl_pid=0, nl_groups=00000000}, msg_namelen=12, msg_iov=[{iov_base=[{{len=72, type=0x14 /* NLMSG_??? */, flags=NLM_F_MULTI, seq=1513194538, pid=25384}, "\n\200\200\376\1\0\0\0\24\0\1\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\1\24\0\6\0"...}, {{len=72, type=0x14 /* NLMSG_??? */, flags=NLM_F_MULTI, seq=1513194538, pid=25384}, "\n@\200\375\2\0\0\0\24\0\1\0\376\200\0\0\0\0\0\0\2\r\271\377\3765:@\24\0\6\0"...}, {{len=72, type=0x14 /* NLMSG_??? */, flags=NLM_F_MULTI, seq=1513194538, pid=25384}, "\n@\200\375\3\0\0\0\24\0\1\0\376\200\0\0\0\0\0\0\2\r\271\377\3765:A\24\0\6\0"...}, {{len=72, type=0x14 /* NLMSG_??? */, flags=NLM_F_MULTI, seq=1513194538, pid=25384}, "\n@\200\375\271\1\0\0\24\0\1\0\376\200\0\0\0\0\0\0\216l\342-s>\364\265\24\0\6\0"...}, {{len=72, type=0x14 /* NLMSG_??? */, flags=NLM_F_MULTI, seq=1513194538, pid=25384}, "\n8\200\0\272\1\0\0\24\0\1\0\375\344\215\272\202\341 \0\0\0\0\0\0\0\0\0\24\0\6\0"...}, {{len=72, type=0x14 /* NLMSG_??? */, flags=NLM_F_MULTI, seq=1513194538, pid=25384}, "\n@\200\375\272\1\0\0\24\0\1\0\376\200\0\0\0\0\0\0A@B1l\317\247\232\24\0\6\0"...}, {{len=131092, type=0x842a /* NLMSG_??? */, flags=NLM_F_REQUEST|NLM_F_DUMP_INTR|NLM_F_DUMP_FILTERED|0x5a00, seq=25384, pid=8460290}, "\3\0\0\0\10\0\1\0\n**\4\10\0\2\0\n**\4\10\0\4\0\n**\377\v\0\3\0"...}], iov_len=4096}], msg_iovlen=1, msg_controllen=0, msg_flags=0}, 0) = 432 recvmsg(7, {msg_name={sa_family=AF_NETLINK, nl_pid=0, nl_groups=00000000}, msg_namelen=12, msg_iov=[{iov_base=[{{len=20, type=NLMSG_DONE, flags=NLM_F_MULTI, seq=1513194538, pid=25384}, "\0\0\0\0"}, {{len=1, type=0x14 /* NLMSG_??? */, flags=NLM_F_REQUEST, seq=0, pid=0}}], iov_len=4096}], msg_iovlen=1, msg_controllen=0, msg_flags=0}, 0) = 20 close(7) = 0
I see no big differences. I attach the full strace log.
Updated by pespin over 6 years ago
On the Prod machine, I checked how other tools set the IP (successfully).
I can create the tunnel with:
ip tuntap add dev tun6 mode tun user root group root
ip addr uses the entlink interface to set the ip.
ifconfig uses the same bits as us:
ioctl(5, SIOCGIFINDEX, {ifr_name="tun6", ifr_index=56}) = 0 ioctl(5, SIOCSIFADDR, {ifr_name="<FD>䍺<82><E1> ", ifr_addr={0x38 /* AF_??? */, "\0\0008\0\0\0\10\5\223\366\332\177\0\0"}}) = 0
I am probably not seeing the interface being created by osmo-ggsn because it is destroyed after trying to fetch the link-local ip, since apn_stop() is called which in turn calls tun_free().
Thinking more about it, that's what I think could be happening: IIRC, setting the IP is not immediate/synchronous, specially because some routing entries are added automatically. Which means we are probably setting up the tunnel correctly, but when we call getifaddrs() to get the list of IPs, the IP we just set is not yet there and we fail to get it. Then we destroy the iface.
We could try waiting a bit more or using the debugger to stop it for a while and see what happens (and if the iface is indeed correctly created and the IP is set).
Updated by pespin over 6 years ago
With gdb I can see that the interface is created and that the IP is assigned. However, no link-local IP is assigned to the interface.
Prod:
62: tun46: <POINTOPOINT,UP,LOWER_UP> mtu 1500 qlen 500 inet6 fde4:8dba:82e1:2000::/56 scope global valid_lft forever preferred_lft foreve
R&D:
447: tun46: <POINTOPOINT,UP,LOWER_UP> mtu 1500 state UNKNOWN qlen 500 inet6 fde4:8dba:82e1:2000::/56 scope global valid_lft forever preferred_lft forever inet6 fe80::1d0:8ef4:21ff:fcc4/64 scope link flags 800 valid_lft forever preferred_lft forever
I'm not sure how is that link-local IP assigned though.
Updated by pespin over 6 years ago
- Status changed from New to Feedback
- % Done changed from 0 to 80
I added code to osmo-ggsn to manually set up a link-local ip.
It can be used in osmo-gsm-tester as soon as it is merged + the trial is built: https://gerrit.osmocom.org/#/c/5349/
Then, we need to add the following line to osmo-ggsn.cfg in osmo-gsm-tester, for inet6:
ipv6 link-local fe80::1111:1111:1111:1111/64
and to inet64:
ipv6 link-local fe80::1111:1111:1111:1112/64
After this options are used, osmo-ggsn tester should start and configure ipv6 pools correctly, and we can enable ipv6 ctx tests again.
Updated by pespin over 6 years ago
- Status changed from Feedback to Resolved
- % Done changed from 80 to 100
osmo-ggsn patches were merged. I pushed just now a commit for osmo-gsm-tester in gerrit to use the option. I already tried in a branch running in Prod setup that it works fine and that ipv6 context can be created successfully using ofono+osmo-ggsn.