VAMOS is a relatively new/late extension of GSM to increase capacity without adding additional transceivers to BTS.
It does this by running two calls (by two different phones/subscribers) on the same ARFCN/timeslot at the same time. The two calls are only distinguished by their training sequence (part of the midamble of each burst).
On the transmit side, the BTS combines the symbols for both subscribers into one stream of symbols. On the receive side, separating the two signals is hard, as the signals are not orthogonal and one can really only correlate on the training sequence in the midanble, and then use that to estimate the channel and recover one signal, and later the second.
There's a nice summary of VAMOS available in this Rohde & Schwarz paper
Unfortunately, 3GPP has only specified the Um interface related bits about VAMOS, but did not specify how e.g. Abis OML / RSL shall be modified to accommodate VAMOS in a standardized way.