Project

General

Profile

Feature #3618

Inter-MSC hand-over support

Added by laforge 9 months ago. Updated about 2 months ago.

Status:
In Progress
Priority:
High
Assignee:
Category:
-
Target version:
-
Start date:
10/02/2018
Due date:
% Done:

90%

Resolution:
Tags:

Description

This is a high-level task for supporting inter-MSC hand-over in OsmoMSC. See InterMSC_Handover for more technical details.

We'll collect sub-tasks here for individual work items.

first_inter_msc_handover_cfg_log.tgz first_inter_msc_handover_cfg_log.tgz 137 KB cfg and log files of first working inter-MSC handover neels, 04/07/2019 01:03 AM
first_inter_msc_handover.pcapng.gz first_inter_msc_handover.pcapng.gz 751 KB gzipped pcap of first working inter-MSC handover neels, 04/07/2019 01:03 AM
multiple_subsequent_inter_msc_handovers.pcapng.gz multiple_subsequent_inter_msc_handovers.pcapng.gz 1.77 MB neels, 04/09/2019 12:11 AM
dtap_during_inter_msc_handover.pcapng.gz dtap_during_inter_msc_handover.pcapng.gz 2.74 MB neels, 04/09/2019 12:21 AM

Subtasks

Feature #3619: GSUP protocol extension for inter-MSC HOClosedneels

Feature #3620: Implement TTCN-3 testsuite for Inter-MSC HOIn Progressneels


Related issues

Related to OsmoHLR - Bug #3710: hlr_ussd.c hardcodes "MSC-00-00-00-00-00-00" twiceResolved11/26/2018

Related to OsmoMSC - Feature #2487: MSC side of LCLS (local call local switching) as per the 3GPP specsStalled09/03/2017

Related to libosmo-sccp + libosmo-sigtran - Bug #3871: osmo_scu_prim conn_id should be scoped per-user and direction, and should not correlate with the local-reference spoken on the SCCP wireIn Progress03/28/2019

Blocked by OsmoMSC - Feature #1609: Inter-BSC hand-over is missing (MSC side)Resolved11/21/201611/21/2016

Blocked by OsmoMSC - Bug #3355: OsmoMSC doesn't provide unique IDTAG_SERNR in IPA CCMResolved06/23/2018

History

#1 Updated by laforge 9 months ago

  • Blocked by Feature #1609: Inter-BSC hand-over is missing (MSC side) added

#2 Updated by laforge 9 months ago

  • Assignee set to laforge
  • Priority changed from Normal to High

#4 Updated by neels 7 months ago

  • Blocked by Bug #3355: OsmoMSC doesn't provide unique IDTAG_SERNR in IPA CCM added

#5 Updated by neels 7 months ago

  • Related to Bug #3710: hlr_ussd.c hardcodes "MSC-00-00-00-00-00-00" twice added

#6 Updated by stsp 6 months ago

This patch adds VTY commands to manage a neighbor list:
https://gerrit.osmocom.org/c/osmo-msc/+/12446

#7 Updated by laforge 6 months ago

  • Assignee changed from laforge to neels

#8 Updated by msuraev 5 months ago

  • Related to Feature #2487: MSC side of LCLS (local call local switching) as per the 3GPP specs added

#9 Updated by neels 5 months ago

  • Status changed from New to In Progress
  • % Done changed from 0 to 30

#10 Updated by neels 4 months ago

  • % Done changed from 30 to 60

I have emerged from an extensive period of osmo-msc refactoring.
The purpose is to separate the RAN connection code from the MSC subscriber handling, to be able to separate both to two distinct osmo-msc instances.
3GPP TS 49.008 4.3 defines the roles of MSC-A, MSC-I and MSC-T, which are now reflected in the osmo-msc code base (branch neels/ho).

The current branch status passes all ttcn3-msc-test test cases that current master passes, as well as two more currently marked as XFail.
However, some TTCN3 tests need fixing to achieve this. The reasons:

  • The tests require erratic Paging semantics. The BSC's job is to periodically repeat Paging, the MSC should do no such thing.
    Hence the tests should not check for "exactly 10" Pagings from osmo-msc, but expect exactly one Paging Request.
    The 10 repeated Pagings are due to odd sms_queue.c behavior, which triggers the queue to collect SMS from the DB for any MT SMS failure.
    Especially when only one SMS is pending in the database, this repeatedly selects the same SMS.
    So the 10 repeated Pagings are actually an sms_queue bug of inadequate shut-upping after failed MT SMS.
  • In some cases, I fixed osmo-msc to send a CC and MNCC Release before disconnecting.
    ttcn3-msc-tests need to expect / ignore these messages to not fail.
  • All voice call tests so far send Assignment Complete messages that lack a Speech Codec (Chosen) IE.
    The new osmo-msc rejects these messages properly, while the current osmo-msc seems to continue nevertheless.
    Need to add Speech Codec (Chosen) IE.
  • In some cases, the ttcn3 code relies on the exact sequence of CRCX, MDCX and Assignment messages,
    which are more flexible in the new osmo-msc (happen as soon as required information becomes available,
    instead of in a hardcoded sequence). Tests need to "interleave" more for that.
  • In some cases the tests wait for only one DLCX, while the new osmo-msc already has two CI set up to DLCX.
    Need to extend to two DLCX. (Best would be either one or two DLCX, will see about that)
  • Some tests send an MT SMS, and then leave it behind in the SMS database. It then shows up in later unrelated tests
    and, for example, clutters the "Clear" phase of a Location Updating.
    The only reason the current tests don't choke on that SMS is above mentioned odd sms_queue behavior.
    IIUC the SMS is then still in the queue having reached some maximum retries count, vs. having been tried once.
    After fixing the sms_queue, the tests need to clear pending SMS -- for which osmo-msc will provide a VTY command.

Ongoing work will now be:

  • implement the inter-BSC handover messages; I am expecting no surprises there.
    • stsp has already prepared neighbor configuration commands for this.
    • especially since I've recently implemented the osmo-bsc side of this, I hope to be done with this quickly.
  • implement the remote-MSC-{A,I,T} roles that forward via GSUP.
    • this is essentially the same as inter-BSC HO, only with MSC-role implementations that forward events instead of directly dispatching.
    • osmith has already prepared GSUP forwarding for this.

As I coined the term in the sysmocom meetings, the first 10% of this job took 90% of the time: preparing osmo-msc for split MSC roles.
I am now implementing the remaining 90%, which I expect to be comparatively quick.
The biggest unknowns here:
- call forwarding via MNCC between remote MSC-A and MSC-I roles.
- how well our GSUP messaging will in fact translate to MAP and interop with other brands of MSC
(but to clarify, I don't consider translation to MAP and interop testing part of my task; apart from maybe adjusting GSUP to allow easy MAP translation).

#11 Updated by neels 4 months ago

What I forgot to mention:

The ttcn3-msc-tests all pass, but also I have tested

  • with a physical BTS and verified all of SMS, USSD and voice calls,
  • with a physical 3G femto cell and verified all of SMS, USSD and voice calls there as well.

This is quite satisfying to me, because I am now using a far stronger abstraction from BSSAP and RANAP, which is one requirement for distributing the MSC roles.
The entire SCCP stack and the entire subscriber management is now essentially a single code path -- with separate message encoding and decoding on the side.
To see that all of this is now working just as, if not better than, it always did before the refactoring is a pretty cool milestone achievement.

Now on to implementing the new features and to prove that all of this was worth the time and effort.

#12 Updated by laforge 4 months ago

Hi Neels,

congratulations to this big achievement. I'm very happy to hear you're past this point of
refactoring, and even more so that you're happy with the result :)

#13 Updated by neels 3 months ago

  • % Done changed from 60 to 70

The largest part of this work was getting to this point: the refactorings pass the TTCN test suite, and now inter-BSC HO has worked for the first time.
What remains now is to implement forwarding of the same inter-BSC HO procedures across a GSUP link to make it an inter-MSC HO.
The largest remaining uncertainty is the call forwarding via MNCC between the two MSCs -- and of course interop with 3rd party MSC vendors.

#14 Updated by laforge 3 months ago

Hi Neels,

On Thu, Mar 14, 2019 at 03:07:14AM +0000, neels [REDMINE] wrote:

The largest remaining uncertainty is the call forwarding via MNCC between the two MSCs --

What's uncertain about it? I do realize it's a bunch of work, but what are the uncertainties?

Did you make something like a message sequence chart showing the BSSAP,
GSUP (or MAP) and SIP (or ISUP) messages during the inter-msc handover
caes? If yes, I think it should be rather easy to expand from there
and add the MNCC primitives that lead to the related SIP/ISUP
transactions in PSTN. Let me know if there's something I can help in
reviewing/expanding.

[...] and of course interop with 3rd party MSC vendors.

that part we cannot test in the Osmocom community as we don't have a GSUP-MAP gateway. This will be up to
whatever party implementing such a gateway.

#15 Updated by neels 3 months ago

Maybe in general uncertainty was the wrong word, more like open tasks where I'm not quite sure that I know what I'm doing yet.

laforge wrote:

The largest remaining uncertainty is the call forwarding via MNCC between the two MSCs --

What's uncertain about it?

  • how I will achieve it in osmo-msc's MNCC code (implementation details, i.e. just a bunch of work)
  • how I need to configure the PBX for testing: resolving the Handover Number MSISDN / picking a useful MSISDN / configuring osmo-msc to pick a useful MSISDN for HO
  • codec negotiation, as we lack proper SDP style codec in MNCC. It should work with hard-coded codecs, and offload all transcoding to the PBX if required, but that's a topic I personally don't know much about. I do know whom to ask though, e.g. whyteks and Thomas from 35c3 POC. Should work out in the end.

[...] and of course interop with 3rd party MSC vendors.

that part we cannot test in the Osmocom community as we don't have a GSUP-MAP gateway. This will be up to
whatever party implementing such a gateway.

yes, am aware, but what information we might lack to facilitate interop is an uncertainty, which will of course be resolved as soon as whoever implements interop needs it.

#16 Updated by neels 3 months ago

  • Related to Bug #3871: osmo_scu_prim conn_id should be scoped per-user and direction, and should not correlate with the local-reference spoken on the SCCP wire added

#17 Updated by neels 3 months ago

  • % Done changed from 70 to 80

I am ever so close. The first inter-MSC handover of the lchan has succeeded, now only need to polish out the almost finished MNCC forwarding code to get voice forwarded properly.

#18 Updated by neels 3 months ago

The first ever inter-MSC Handover with working voice forwarding has happened just now.
It feels historic, so I'm posting a pcap of it, even though there are most probably plenty open details.
Also attaching the config files and logs.

The wireshark patch attached to https://osmocom.org/issues/3893 allows showing most of the inter-MSC GSUP messages in a nice way. ( https://osmocom.org/attachments/3626/0001-GSUP-add-E-interface-messages-wip.patch )

Using these branches:

             asn1c master
          libasn1c master
      libosmo-abis master
     libosmo-netif master
      libosmo-sccp neels/conn_id
       libosmocore neels/gsup_router
         libsmpp34 master
          osmo-bsc neels/ho-osmo-bsc
          osmo-hlr neels/gsup_router
          osmo-iuh master
          osmo-mgw neels/endpoint_fsm
          osmo-msc neels/ho
osmo-sip-connector master

#19 Updated by neels 3 months ago

Attaching a pcap with multiple successive inter-MSC handovers of the same call.

(In the pcap, the RTP stream goes directly from osmo-mgw to osmo-mgw, which is only because kamailio tells osmo-msc the RTP address of the other osmo-msc directly in the MNCC_RTP_CONNECT messages)

I'm running out of problems that need fixing, so will concentrate on applying gerrit code review to preparatory patches.
ttcn3 testing could also receive more attention.

#20 Updated by neels 3 months ago

Attaching a pcap of successful SMS negotiation during inter-MSC handover, i.e. DTAP via a remote MSC.

#21 Updated by neels about 2 months ago

The patches to implement inter-MSC Handover are now merged to libosmocore, osmo-hlr and osmo-msc master branches.

The osmo-msc.cfg for inter-MSC Handover however still has this open issue:
The handover-number config so far uses a number range that does not support leading zeros.
This should change to something like

 handover-number range 001234xxx
 handover-number range 001234xxx FIRST LAST  (if desired to limit the 'xxx' digits to a specific range)

#22 Updated by neels about 2 months ago

To summarize, osmo-msc master is ready for testing inter-MSC (and inter-BSC) handover.
The config for the handover number range is likely to still change before the next official release (trivial change to config files format compared to current master).

Also available in: Atom PDF

Add picture from clipboard (Maximum size: 48.8 MB)