https://projects.osmocom.org/https://projects.osmocom.org/favicon.ico?16647414092018-10-03T08:58:13ZOpen Source Mobile CommunicationsOsmoSTP - Feature #3608: Support for SCTP multi-hominghttps://projects.osmocom.org/issues/3608?journal_id=119282018-10-03T08:58:13Zlaforge
<ul></ul> OsmoSTP - Feature #3608: Support for SCTP multi-hominghttps://projects.osmocom.org/issues/3608?journal_id=122472018-10-16T19:51:04Zmandersen
<ul></ul><p>laforge wrote:</p>
<blockquote>
<p>My humble assumption here is that this is done entirely in the kernel. It will automatically learn about the peers' additional addresses and<br />use them as needed.</p>
</blockquote>
<p>In my experience the kernel will need a bit of help before the multi-homing works as we expect it to.</p>
<p>Consider a scenario where the STP runs on a server with 4 IPs (A, B, C and D). A and B are used for signaling, but C and D are firewalled and used for OAM/local SSH logins.</p>
<p>If we open a multi-homed SCTP connection to another host with kernel defaults, then the kernel (not knowing about our overall network design) will associate all 4 IPs (A, B, C and D) with this connection.</p>
<p>As a consequence, the peer will successfully communicate with our IPs A and B - but endless try (and fail) to communicate with our IPs C and D.</p>
<p>Therefore it can make sense to explicitly specify:<br />1. Which peer IPs this SCTP association connects to<br />2. Which IPs we bind on our end of the SCTP association</p>
<p>This explicit binding of IPs on our side can done by using sctp_bindx() on the fd before calling sctp_connectx().</p> OsmoSTP - Feature #3608: Support for SCTP multi-hominghttps://projects.osmocom.org/issues/3608?journal_id=122482018-10-16T20:30:08Zlaforge
<ul></ul><p>On Tue, Oct 16, 2018 at 07:51:04PM +0000, mandersen [REDMINE] wrote:</p>
<blockquote>
<p>Issue <a class="issue tracker-2 status-5 priority-2 priority-default closed" title="Feature: Support for SCTP multi-homing (Closed)" href="https://projects.osmocom.org/issues/3608">#3608</a> has been updated by mandersen.</p>
</blockquote>
<p>thanks a lot for your most valuable input.</p> OsmoSTP - Feature #3608: Support for SCTP multi-hominghttps://projects.osmocom.org/issues/3608?journal_id=161752019-10-08T14:55:32Zlaforge
<ul><li><strong>Assignee</strong> set to <i>pespin</i></li></ul> OsmoSTP - Feature #3608: Support for SCTP multi-hominghttps://projects.osmocom.org/issues/3608?journal_id=161762019-10-08T14:57:31Zlaforge
<ul><li><b>Checklist item</b> <input type='checkbox' class='checklist-checkbox' disabled> <i>TTCN-3 test cases</i> added</li><li><b>Checklist item</b> <input type='checkbox' class='checklist-checkbox' disabled> <i>implementation in libosmo-sccp/osmo-stp</i> added</li></ul>The SCTP protocol provides built-in functionality for multi-homing, that is multiple IP addresses<br />on either side of the SCTP association. This functionality is present in the Linux kernel SCTP<br />implementation, which in turn is used by osmo-stp. However, osmo-stp currently does not provide<br />any means to configure which of the hosts IP adresses are used for what particular AS or ASP.<br />The implementation of this feature will include
<ol>
<li>Extension of the VTY interface to configure multiple local and remote IP adresses</li>
<li>Extension of the osmo-stp internal data structures to handle multiple IP adresses</li>
<li>Corresponding configuration of the Linux kernel SCTP stack using its socket API from osmo-<br />stp
<ol>
<li>local IP addresses are used for the sctp_bindx() socket API call</li>
<li>remote IP addresses are used for the sctp_connectx() socket API call</li>
</ol>
</li>
<li>Implementation of automatic test cases in TTCN-3</li>
</ol> OsmoSTP - Feature #3608: Support for SCTP multi-hominghttps://projects.osmocom.org/issues/3608?journal_id=161902019-10-09T17:15:06Zpespin
<ul><li><strong>Status</strong> changed from <i>New</i> to <i>In Progress</i></li></ul> OsmoSTP - Feature #3608: Support for SCTP multi-hominghttps://projects.osmocom.org/issues/3608?journal_id=161932019-10-09T18:27:21Ztna-signaling
<ul></ul><p>For point 3-1, please limit the local IP addresses to those actually some SCTP application. <br />One of our vendors actually has this bug (never fixed) where sctp_bindx() picks up all the local addresses (including localhost, etc) that do not service SCTP, and that messes up multi-homing.</p> OsmoSTP - Feature #3608: Support for SCTP multi-hominghttps://projects.osmocom.org/issues/3608?journal_id=161942019-10-09T18:55:32Zpespin
<ul></ul><p>In osmo-* programs, SCTP socket is handled over osmo_stream_*.</p>
<p>We are using one-to-one style socket paradigm there afaict (<a class="external" href="https://tools.ietf.org/html/draft-ietf-tsvwg-sctpsocket-15#section-4.1.1">https://tools.ietf.org/html/draft-ietf-tsvwg-sctpsocket-15#section-4.1.1</a>).</p>
<p>SCTP osmo_stream_srv_link is configured in osmo_ss7_xua_server_create(), and osmo_ss7_xua_server_bind() calls osmo_stream_srv_open(), which in turn calls osmo_sock_init(AF_INET, SOCK_STREAM, link->proto,... (link->proto=IPPROTO_SCTP), which creates the socket() and calls listen() (meaning new SCTP associations can be done at this point).<br />SCTP osmo_stream_cli is configured in libosmo-sccp.git osmo_ss7_asp_restart(). osmo_stream_cli_open2() calls osmo_sock_init2(AF_INET, SOCK_STREAM, cli->proto, ... (cli->proto=IPPROTO_SCTP)<br />SCTP osmo_stream_srv i processed through xua_srv_conn_cb() in libosmo-sccp. In there there's a call to sctp_recvmsg(). Sending in done in osmo_ss7_asp_send() which calls osmo_stream_srv_send() and enqueue the msg, and eventually osmo_stream_cli_write() will call sctp_send().<br />In this later case, we do stuff with SCTP_ASSOC_CHANGE and other notifications.</p>
<p>SCTP send() is in osmo_stream_srv_write()<br />SCTP sctp_recvmsg() is in osmo_stream_srv_recv()</p>
<p>As per what I understand so far, dummy multi-homing is already supported right now in the generic socket API (in clients like BSC and MSC, not sure about servers):<br /><a class="external" href="https://tools.ietf.org/html/draft-ietf-tsvwg-sctpsocket-15#section-4.1.2">https://tools.ietf.org/html/draft-ietf-tsvwg-sctpsocket-15#section-4.1.2</a><br />""" <br /> If addr is specified as a wildcard (INADDR_ANY for an IPv4 address,<br /> or as IN6ADDR_ANY_INIT or in6addr_any for an IPv6 address), the<br /> operating system will associate the endpoint with an optimal address<br /> set of the available interfaces.<br />"""</p>
<p>Similary for connect (we always do bind() before thought, but in case it's bind(ANY_ADDR):<br /><a class="external" href="https://tools.ietf.org/html/draft-ietf-tsvwg-sctpsocket-15#section-4.1.5">https://tools.ietf.org/html/draft-ietf-tsvwg-sctpsocket-15#section-4.1.5</a><br />""" <br /> If a bind() is not called prior to the connect() call, the system<br /> picks an ephemeral port and will choose an address set equivalent to<br /> binding with INADDR_ANY and IN6ADDR_ANY for IPv4 and IPv6 socket<br /> respectively. One of those addresses will be the primary address for</p>
<p>"""</p>
<p>In osmo-bsc/osmo-msc (osmo_stream_cli), if no "local-ip" under "asp" node is used ("0.0.0.0"), the above will happen.<br />In osmo-stp (osmo_stream_srv), if no "local-ip" under "listen" node is used (aka "0.0.0.0"), the above will happen.<br />With that config, I can see in my case both the STP and the BSC/MSC announcing all the IP addrs on my system and later on attempting HEARTBEATS for each of them.</p>
<p>So the new APIs are really needed to create specifics sets for local addresses (bind). From VTY user point of view, simply allowing several "local-ip" cmds (first one being primary) and "no local-ip" to remove them should be sufficient. Maybe adding an optional parameter "[primary]" at the end of "local-ip" cmd.</p>
<p>Regarding decision on who to send:<br /><a class="external" href="https://tools.ietf.org/html/draft-ietf-tsvwg-sctpsocket-15#section-4.1.8">https://tools.ietf.org/html/draft-ietf-tsvwg-sctpsocket-15#section-4.1.8</a><br />""" <br />[msg_name] is used to indicate a<br /> preferred peer address if the sender wishes to discourage the stack<br /> from sending the message to the primary address of the receiver. If<br /> the transport address given is not part of the current association,<br /> the data will not be sent and a SCTP_SEND_FAILED event will be<br /> delivered to the application if send failure events are enabled.<br />""" <br />So in theory we should have a "[primary]" option in "remote-ip" too, or even some param specific order 1...N in order to decide easily which remote IP addr to send messages. That'd be used when sctp_connectx() also.</p>
<p>I still need to read part of the socket API to better find what and how I need to implement it (I'm at point "5.2. SCTP msg_control Structures" right now).</p> OsmoSTP - Feature #3608: Support for SCTP multi-hominghttps://projects.osmocom.org/issues/3608?journal_id=161952019-10-10T11:04:43Zpespin
<ul></ul><p>It seems ip addr list in sctp_sendx() is actually only used for implicit connection if no connect() was called before it, so we cannot really specifyat that time which dst addr to use. Morevoer, specs says:<br /><a class="external" href="https://tools.ietf.org/html/draft-ietf-tsvwg-sctpsocket-15#section-8.11">https://tools.ietf.org/html/draft-ietf-tsvwg-sctpsocket-15#section-8.11</a><br /><pre>
The list of addresses is provided for implicit
association setup. In such a case the list of addresses serves the
same purpose as the addresses given in sctp_connectx (see
Section 8.9).
</pre></p>
<p>This is also interesting:<br /><pre>
The system uses stream 0 as the default
stream for send() and sendto(). recv() and recvfrom() return data
from any stream, but the caller can not distinguish the different
streams.
</pre></p>
<p>There's also:<br /> <a class="external" href="https://tools.ietf.org/html/draft-ietf-tsvwg-sctpsocket-15#section-7.1.10">https://tools.ietf.org/html/draft-ietf-tsvwg-sctpsocket-15#section-7.1.10</a><br /><pre>
7.1.10. Set Primary Address (SCTP_PRIMARY_ADDR)
Requests that the local SCTP stack use the enclosed peer address as
the association primary. The enclosed address must be one of the
association peer's addresses.
</pre></p>
<p>So after all what we need is probably accepting multiple "local-ip" and "remote-ip", and perhaps adding an optional "[primary]" cmd to be able to use SCTP_PRIMARY_ADDR on that IP addr. If no primary param is provided, I'd expect the first "remote-ip" line (and first in array to connectx()) to be stream 0 and thus the primary one.</p> OsmoSTP - Feature #3608: Support for SCTP multi-hominghttps://projects.osmocom.org/issues/3608?journal_id=162002019-10-10T19:00:51Zpespin
<ul></ul><p>I have been working on libosmocore's API osmo_sock_init2_multiaddr() to create the socket with multiple local and remote addresses. I also added support for it in libosmo-netif's osmo_stream_server_link*() APIs.</p>
<p>Now I'm looking at libosmo-sccp.git. First problem while trying to implement the "local-ip" set in server VTY. For instance:<br /><pre>
listen m3ua 2905
accept-asp-connections dynamic-permitted
local-ip 192.168.30.1
local-ip 192.168.30.1
</pre></p>
<ul>
<li>"listen" vty node calls "osmo_ss7_xua_server_create()" which creates xua server with host=NULL (0.0.0.0)</li>
<li>Right now "local-ip" calls osmo_ss7_xua_server_bind()->osmo_stream_srv_link_open() every time, so it's impossible to set more than one local-ip. That bind needs to be removed and be done later. There's 2 possibilities:
<ul>
<li>let the main.c in osmo-stp do the right thing. It already does thanks to a commit I did a while ago in libosmo-sccp.git (10d4815bb1b4b548ec0bc97611b2e7ac45e0ebc5). But other applications may break maybe because they don't call that API introduced in the commit?</li>
<li>Apply bind() upon exit to the node. That seems to be the cleanest option imho, but requires the feature of "exit on node callback" be added to libosmocore.</li>
</ul></li>
</ul>
<p>Right now with the new osmo_sock_init2_multiaddr() which calls sctp_bindx binding to one specific IP (12.7.0.0.1 in this case) already seems to change the beahavor, that is, osmo-stp doesn't bind+listen to all the IP addr:<br /><pre>
$ cat /proc/net/sctp/assocs
ASSOC SOCK STY SST ST HBKT ASSOC-ID TX_QUEUE RX_QUEUE UID INODE LPORT RPORT LADDRS <-> RADDRS HBINT INS OUTS MAXRT T1X T2X RTXC wmema wmemq sndbuf rcvbuf
4014cd21 71236246 2 1 3 0 22 0 0 1000 761112 2905 44660 127.0.0.1 <-> *127.0.0.1 127.0.0.2 192.168.30.1 192.168.30.100 192.168.1.132 192.168.1.134 10.8.0.11 9000 10 10 10 0 0 0 1 0 212992 212992
2ded729f 12c601ec 2 1 3 0 21 0 0 1000 759276 44660 2905 127.0.0.1 127.0.0.2 192.168.30.1 192.168.30.100 192.168.1.132 192.168.1.134 10.8.0.11 <-> *127.0.0.1 9000 10 10 10 0 0 0 1 0 212992 212992
b2d3c479 3c936823 2 1 3 0 19 0 0 1000 755464 58089 2905 127.0.0.1 127.0.0.2 192.168.30.1 192.168.30.100 192.168.1.132 192.168.1.134 10.8.0.11 <-> *127.0.0.1 9000 10 10 10 0 0 0 1 0 212992 212992
8dbe9932 680b97d8 2 1 3 0 20 0 0 1000 752381 2905 58089 127.0.0.1 <-> *127.0.0.1 127.0.0.2 192.168.30.1 192.168.30.100 192.168.1.132 192.168.1.134 10.8.0.11 9000 10 10 10 0 0 0 1 0 212992 212992
</pre></p>
<p>And here I passed a manually handcrafted ip addr from my system (192.168.30.1) into the code in libosmo-sccp (because VTY support is not yet there), so local binding seems to be working so far:<br /><pre>
$ cat /proc/net/sctp/assocs
ASSOC SOCK STY SST ST HBKT ASSOC-ID TX_QUEUE RX_QUEUE UID INODE LPORT RPORT LADDRS <-> RADDRS HBINT INS OUTS MAXRT T1X T2X RTXC wmema wmemq sndbuf rcvbuf
46a3bdc3 a349c970 2 1 3 0 47 0 0 1000 901729 41993 2905 127.0.0.1 127.0.0.2 192.168.30.1 192.168.30.100 192.168.1.132 192.168.1.134 10.8.0.11 <-> *127.0.0.1 192.168.30.1 9000 10 10 10 0 0 0 1 0 212992 212992
f123ce70 4f7ce2c8 2 1 3 0 45 0 0 1000 903322 36889 2905 127.0.0.1 127.0.0.2 192.168.30.1 192.168.30.100 192.168.1.132 192.168.1.134 10.8.0.11 <-> *127.0.0.1 192.168.30.1 9000 10 10 10 0 0 0 1 0 212992 212992
f8c62b1b adbb8780 2 1 3 0 48 0 0 1000 897001 2905 41993 127.0.0.1 192.168.30.1 <-> *127.0.0.1 127.0.0.2 192.168.30.1 192.168.30.100 192.168.1.132 192.168.1.134 10.8.0.11 9000 10 10 10 0 0 0 1 0 212992 212992
35c6ed4d 651b3248 2 1 3 0 46 0 0 1000 904313 2905 36889 127.0.0.1 192.168.30.1 <-> *127.0.0.1 127.0.0.2 192.168.30.1 192.168.30.100 192.168.1.132 192.168.1.134 10.8.0.11 9000 10 10 10 0 0 0 1 0 212992 212992
4601930b c4c33896 2 1 3 0 44 0 0 1000 903314 2905 38603 127.0.0.1 192.168.30.1 <-> *127.0.0.1 127.0.0.2 192.168.30.1 192.168.30.100 192.168.1.132 192.168.1.134 10.8.0.11 9000 10 10 10 0 0 0 1 0 212992 212992
223db1ca b3b084e1 2 1 3 0 43 0 0 1000 904311 38603 2905 127.0.0.1 127.0.0.2 192.168.30.1 192.168.30.100 192.168.1.132 192.168.1.134 10.8.0.11 <-> *127.0.0.1 192.168.30.1 9000 10 10 10 0 0 0 1 0 212992 212992
</pre></p> OsmoSTP - Feature #3608: Support for SCTP multi-hominghttps://projects.osmocom.org/issues/3608?journal_id=162092019-10-11T12:23:08Zpespin
<ul></ul><p>I finally resolved the VTY issue by making sure the local-ip is bound not during cmd time but upon VTY node exits (so we can record several "local-ip" cmds, then apply all of them).</p>
<p>While doing so, I was hit by a libosmovty bug about go_parent_cb not called at the end of a file which I solved here:<br /><a class="external" href="https://gerrit.osmocom.org/c/libosmocore/+/15777">https://gerrit.osmocom.org/c/libosmocore/+/15777</a> vty: Fix go_parent_cb not called for indented nodes at end of cfg file</p> OsmoSTP - Feature #3608: Support for SCTP multi-hominghttps://projects.osmocom.org/issues/3608?journal_id=162132019-10-11T19:38:27Zpespin
<ul></ul><p>I submitted a first version implementing the desired features. I did some manual tests with osmo-bsc, osmo-msc and osmo-stp talking each other with different configurations and looks fine as far as I can tell so far.</p>
<p>With these changes, several "local-ip" and "remote-ip" lines are accepted under "listen" and "asp" VTY nodes, allowing to configure an SCTP connection with multiple connections, hence allowing control of SCTP multi-homing features. libosmo-sccp clients such as osmo-bsc and osmo-msc also gain support for this feature with these changes.</p>
<p>libosmocore (I still need to fix build for embedded targets):<br /><a class="external" href="https://gerrit.osmocom.org/c/libosmocore/+/15781">https://gerrit.osmocom.org/c/libosmocore/+/15781</a> socket: Introduce API osmo_sock_init2_multiaddr()</p>
<p>libosmo-netif:<br />remote: <a class="external" href="https://gerrit.osmocom.org/c/libosmo-netif/+/15782">https://gerrit.osmocom.org/c/libosmo-netif/+/15782</a> stream: osmo_stream_srv_link: Support setting multiple addr<br />remote: <a class="external" href="https://gerrit.osmocom.org/c/libosmo-netif/+/15783">https://gerrit.osmocom.org/c/libosmo-netif/+/15783</a> stream: osmo_stream_cli: Support setting multiple addr</p>
<p>libosmo-sccp/osmo-stp:<br />remote: <a class="external" href="https://gerrit.osmocom.org/c/libosmo-sccp/+/15784">https://gerrit.osmocom.org/c/libosmo-sccp/+/15784</a> Defer xua server binding until exit of VTY node<br />remote: <a class="external" href="https://gerrit.osmocom.org/c/libosmo-sccp/+/15785">https://gerrit.osmocom.org/c/libosmo-sccp/+/15785</a> ss7: Support multiple addresses in SCTP connections<br />remote: <a class="external" href="https://gerrit.osmocom.org/c/libosmo-sccp/+/15786">https://gerrit.osmocom.org/c/libosmo-sccp/+/15786</a> ss7: Log local and remote address set upon ASP restart</p>
<p>So current status is:</p>
<ol>
<li>Extension of the VTY interface to configure multiple local and remote IP adresses <-- <strong>DONE</strong></li>
<li>Extension of the osmo-stp internal data structures to handle multiple IP adresses <-- <strong>DONE</strong></li>
<li>Corresponding configuration of the Linux kernel SCTP stack using its socket API from osmo-stp <-- <strong>DONE (TBD: fix build under embedded env)</strong>
<ol>
<li>local IP addresses are used for the sctp_bindx() socket API call <-- <strong>DONE</strong></li>
<li>remote IP addresses are used for the sctp_connectx() socket API call <-- <strong>DONE</strong></li>
</ol>
</li>
<li>Implementation of automatic test cases in TTCN-3 <-- <strong>TBD</strong></li>
</ol> OsmoSTP - Feature #3608: Support for SCTP multi-hominghttps://projects.osmocom.org/issues/3608?journal_id=163462019-11-05T16:34:59Zlaforge
<ul><li><b>Checklist item</b> <input type='checkbox' class='checklist-checkbox' checked disabled> <i>implementation in libosmo-sccp/osmo-stp</i> set to Done</li><li><strong>% Done</strong> changed from <i>0</i> to <i>80</i></li></ul><p>FYI: all patches in master for some time already. only part missing is automatic tests.</p> OsmoSTP - Feature #3608: Support for SCTP multi-hominghttps://projects.osmocom.org/issues/3608?journal_id=163592019-11-07T19:04:52Zpespin
<ul><li><strong>Status</strong> changed from <i>In Progress</i> to <i>Feedback</i></li></ul><p>FYI that's already being tested in a simple way in python unit tests of libosmo-sccp.git:<br /><a class="external" href="https://git.osmocom.org/libosmo-sccp/tree/tests/vty/vty_test_runner.py#n110">https://git.osmocom.org/libosmo-sccp/tree/tests/vty/vty_test_runner.py#n110</a></p>
<p>Not sure if we want to add more complex tests in TTCN3 for now.</p> OsmoSTP - Feature #3608: Support for SCTP multi-hominghttps://projects.osmocom.org/issues/3608?journal_id=164492019-11-13T16:51:25Zlaforge
<ul><li><b>Checklist item</b> changed from <i>TTCN-3 test cases</i> to <i>test cases</i></li><li><b>Checklist item</b> <input type='checkbox' class='checklist-checkbox' checked disabled> <i>test cases</i> set to Done</li><li><strong>Status</strong> changed from <i>Feedback</i> to <i>Closed</i></li><li><strong>% Done</strong> changed from <i>80</i> to <i>100</i></li></ul><p>We decided that python based tests are more applicable for this feature. important factor is that we do have automatic tets, and they do pass. closing.</p>