Integrating Third Party Media Stack with PJSIP 1.x

Table of Contents

1. Introduction
2. SIP and Media Interaction
   1. Creating a Call/Invite? Session
   2. Starting the Media
   3. Subsequent Offer/Answer?
   4. Call Disconnection
3. Third Party Media Library Integration Strategy
   1. Completely Replace PJMEDIA with Third Party Media Stack
   2. Reusing PJMEDIA's Session Info
   3. Integrating Third Party Media Stack into PJMEDIA

---

Update

This article is only applicable for PJSIP 1.x. For PJSIP 2.x see Integrating Third Party Media Stack with PJSIP 2.x

Introduction

PJSIP consists of several libraries which interaction can be seen from the diagram below.

Below are the components that are related to media stack and SIP stack integration.

PJSUA-LIB:

​PJSUA-LIB is a library which integrates ​PJSIP, ​PJMEDIA, and ​PJNATH libraries into very high level and easy to use API. Due to its nature, it is tightly coupled with all of the libraries so it is considered unfeasible to detach either of the library component from PJSUA-LIB.

PJSIP Libraries:

What we normally address as PJSIP itself is a collection of several SIP libraries.

• PJSIP base library (pjsip-core.lib, which includes the ​pjsip core, the ​transaction layer, and ​dialog management),
• SIP user agent library (​pjsip-ua, which includes invite session management, client registration, call features, etc.), and
• SIP presence and instant messaging library (​pjsip-simple).

The SIP user agent library (pjsip-ua) uses the SDP components of PJMEDIA for negotiating offer/answer.

Media Stack:

The media stack includes ​PJMEDIA and PJMEDIA-CODEC. Depending on the library packaging, it may contain SDP components (message representation, parsing, and SDP offer answer negotiation).

Although the SDP components (​message representation, parsing, and SDP ​offer answer negotiation) may be packaged with PJMEDIA, actually it is independent from PJMEDIA and it can be (and has been) used independently from PJMEDIA. This is because although media can be optional for some application (for example, a server), SDP and SDP negotiation are needed by and tightly coupled to the ​invite session (sip_inv.c), so it should be able to use these parts without having to use PJMEDIA.

Note: in Symbian build system, these SDP parts are packaged on separate static library (as PJSDP library) and removed from PJMEDIA. On GNU build system, both PJMEDIA and PJSDP libraries are built, but the SDP parts are included in PJMEDIA as well, so only PJMEDIA is needed to be linked with the application (since the PJMEDIA library here also contains PJSDP parts).

---

SIP and Media Interaction

Depending on which API abstraction layer being used by application (PJSUA-LIB or PJSIP/PJMEDIA API directly), integration between PJSIP and PJMEDIA happens in either PJSUA-LIB library or in application code. In this integration, there are few places where PJSIP and PJMEDIA interact with each other:

• During call/invite session creation -- we need to specify local SDP when creating invite session.
• After SDP negotiation -- once SDP has been negotiated, we can start our media streaming.
• Anytime thereafter when the invite session requires us to update our offer or answer.
• During call disconnection -- to release media.

These will be discussed in more detail in the following sections.

Creating a Call/Invite? Session

An ​invite session is created by application (or PJSUA-LIB) when we make or receive calls. We need to specify our local SDP when creating the invite session, because SDP negotiation will be done by the invite session [*], and to do this it needs SDP from both local and remote endpoints.

[*] To be precise, the negotiation actually is done by the ​SDP negotiator (part of PJMEDIA/PJSDP library) which will be invoked by the invite session. The motivation to make SDP negotiation as integral part of invite session is to shield application from the complexity of SDP offer/answer session (for example, with late offer/answer, offer/answer quirks related to the use of PRACK, and ability to update session early with UPDATE), while maintaining the flexibility for the application to control when and what offer and answer to be given to remote. Also an invite session always negotiate SDP, regardless whether the application is a user agent or a back to back user agent (and indeed people have used pjsip to build B2BUA).

Below is the pseudo-code for creating a call/invite session. The procedure is relatively similar for both caller and callee. For a working sample, please see the ​simpleua sample application or PJSUA-LIB source code.

create_call()
{
    // Create the dialog
    dlg = pjsip_dlg_create_uac/uas();

    // Create media transport (pair of RTP/RTCP socket) for
    // transmitting/receiving the media from remote endpoint
    media_transport = pjmedia_transport_udp_create();

    // Get the socket address info of the media transport
    // The socket address will be put in SDP c= and m= lines.
    media_sock_info = pjmedia_transport_get_info(media_transport);
   
    // Create local and initial SDP.
    // The easiest way to create SDP is to ask the media endpoint
    // to create an SDP describing local media capability for us.
    sdp = pjmedia_endpt_create_sdp(media_sock_info);

    // Create the invite session, giving it both the dialog
    // and local SDP to be negotiated.
    inv = pjsip_inv_create_uac/uas(dlg, sdp);
}

It may be worthwhile to look what ​pjmedia_endpt_create_sdp() function is doing in detail, so below is the pseudo-code of the function:

pjsip_endpt_create_sdp(media_sock_info)
{
  // Create blank SDP structure
  sdp = alloc(sizeof(pjmedia_sdp_session));

  // Fill in base SDP fields (such as t=, s=, c= lines)
  ..

  // Query list of codecs that we support from the codec manager
  codec_list[] = pjmedia_codec_mgr_enum_codecs();

  // Put each codec information in the SDP
  for each codec in codec_list {
    put codec info in SDP
  }

  // Put media socket address info in SDP c= and m= line 
  sdp->connection_line = media_sock_info->rtp.ip_address;
  sdp->m_line.port = media_sock_info->rtp.port_number;

  return sdp;
}

For further reference, please see:

• ​PJSIP Invite Session object and its callbacks.
• ​PJSIP Dialog object
• ​SDP Negotiator object
• ​SDP representation and parsing
• the PJSIP Developer's Guide PDF (available from ​http://www.pjsip.org/docs.htm) contains details information on the interaction between the invite session and SDP negotiator.
• ​siprtp sample application is a sample application that sends/receives RTP packets without using stream.

Starting the Media

As discussed above, SDP negotiation is done by the ​SDP negotiator object that is invoked internally by ​the invite session. When SDP negotiation has been completed, the invite application (or PJSUA-LIB) will get notification via invite session's on_media_update() callback. SDP negotiation is marked as completed as soon as an offer is answered (either in 200/OK response to INVITE, in 18x response, 200/OK response to UPDATE, or in PRACK exchange).

The ​on_media_update() callback contains status parameter which tells application whether SDP negotiation has been successful (or not). We are normally only interested with the successful status to start our media. Failure status will be handled by the invite session (e.g. to send failure response to remote).

Below is the pseudo-code to implement on_media_update() callback. You can see a working implementation of this function in the ​simpleua sample or pjsua_call_on_media_update() function of pjsua_call.c in PJSUA-LIB.

void on_media_update(inv, status)
{
   if (status != PJ_SUCCESS) {
      // Handle failed negotiation scenario. 
      // Normally we don't need to do anything here
      ...
      return;
   }

   // Retrieve both local and remote active SDP's.
   // Active SDP is the SDP that has been negotiated
   // by SDP negotiator, and the content may differ from
   // the initial content when the SDP's are given to
   // the invite session (for example, codec order may be
   // rearranged, and non-active codecs may be removed
   // altogether).
   local_sdp = pjmedia_sdp_neg_get_active_local(inv->neg);
   remote_sdp = pjmedia_sdp_neg_get_active_remote(inv->neg);

   // Update media based on local and remote SDP
   update_media_channel(local_sdp, remote_sdp);
}

And below is the pseudo-code of update_media_channel() function. You can see a working implementation of this function in pjsua_media_channel_update() of pjsua_media.c in PJSUA-LIB or in ​simpleua sample application.

update_media_channel(local_sdp, remote_sdp)
{
   // Create media session info from both local and remote SDP.
   // The media session contains everything that's needed to create our
   // media stream (codec settings, transport settings (local and remote
   // socket addresses), jitter buffer settings, etc.)
   media_sess_info = pjmedia_session_info_from_sdp(local_sdp, remote_sdp);

   // Handle inactive media
   if (media_sess_info->dir = NONE) {
      // Handle inactive media
      ..
      
      return;
   }

   // Customize media session info if wanted (for example, turn VAD/PLC 
   // on/off, change jitter buffer setting, set RTP/RTCP SSRC, etc).
   ..

   // Create the media session, giving it both the media session info
   // and media transport. We created the media transport earlier when
   // we create the invite session.
   m_session = pjmedia_session_create(media_sess_info, media_transport);

   // Extract audio stream from the media session
   stream_port = pjmedia_session_get_port(m_session, 0);

   // Do something with the audio stream. Here we connect it to the sound
   // device.
   pjmedia_snd_port_connect(sound_device, stream_port);
}

For further reference, please see:

• ​Media session object
• ​simpleua sample application
• ​siprtp sample application is a sample application that sends/receives RTP packets without using stream.

Subsequent Offer/Answer?

The invite session provides additional callbacks related to SDP offer/answer negotiation:

• on_rx_offer(): this callback is called when the session has received a new offer from remote, and it needs a local SDP to be negotiated as SDP answer (by calling pjsip_inv_set_sdp_answer() with the SDP).
• on_create_offer(): this callback is called when the session needs to generate a local offer (for example when it receives incoming re-INVITE without an offer).

Application MUST provide a local SDP to the invite session in both callbacks. The process of getting local SDP is similar to the one when creating a call/invite session above. For a working sample, please see pjsua_call_on_rx_offer() and pjsua_call_on_create_offer() functions in pjsua_call.c file of PJSUA-LIB.

For further reference, please see:

• ​Invite Session callback

Call Disconnection

Call disconnection occurs when the invite session calls on_state_changed() callback of ​pjsip_inv_callback with invite session state equal to PJSIP_INV_STATE_DISCONNECTED. Application should release the media resources allocated for the call on this event.

---

Third Party Media Library Integration Strategy

We have three approaches on how to integrate third party media stack with PJSIP:

1. to only use SDP components of PJMEDIA and nothing else (the SDP messaging, parsing, and offer answer negotiation are still needed since they are tightly coupled with the invite session for the reasons explained above).
2. above, plus to reuse PJMEDIA's session info generation to avoid working with SDP directly.
3. integrating third party stack into PJMEDIA

Each integration approach have their pros and cons which will be discussed below.

---

Completely Replace PJMEDIA with Third Party Media Stack

With this approach, basically we don't link/use PJMEDIA at all, except the SDP components of course. Pros and cons of this approach are:

Pros:

• the separation is cleaner thus may be easier to understand. Basically application MUST NOT use anything from PJMEDIA except <sdp.h> and <sdp_neg.h>
• using this approach is supported, since it does not require modifications to our codes/libraries.

Cons:

• This approach can only be used when application uses PJSIP directly (i.e. not PJSUA-LIB), thus it requires significant more efforts to accomplish tasks which otherwise can be done more easily with PJSUA-LIB. But on the positive note, PJSUA-LIB contains ready to use examples on how to do things so one may just need to copy/paste the code into application's code.
• Since we are not allowed to use PJMEDIA's media session info, we need to manually inspect the SDP fields on both local and remote SDP's to determine the parameters for the media session. This could be a tedious operation.

To use this approach, you need to do the following (more or less):

1. Link with the appropriate library:
   • On Symbian or GNU build system, link the application with PJSDP library and exclude PJMEDIA from the link process.
   • On other build system, you may link with PJMEDIA to get the SDP components linked with application, but do not use anything else.
   • Alternatively you may create a new library which consists of the following files from PJMEDIA directory: errno.c, sdp.c, sdp_cmp.c, and sdp_neg.c
2. Register PJMEDIA's specific error code to PJLIB error code space, using the code snippet below. This normally is done when we call pjmedia_endpt_create(), but we can't call this function since we are not using PJMEDIA of course!

   #include <pjmedia/errno.h>
   
   pj_register_strerror(PJMEDIA_ERRNO_START, PJ_ERRNO_SPACE_SIZE, &pjmedia_strerror);
   

3. You need to build the SDP manually for the invite session (either when creating the call or when the invite session needs updated offer or answer). You cannot use pjmedia_endpt_create_sdp() function since this function is part of PJMEDIA.
4. To create a media session, you need to manually inspect both local and remote SDP (see on_media_update() pseudo-code above) and create your media session based on the information in both SDP's.

---

Reusing PJMEDIA's Session Info

The idea of this approach is to allow PJMEDIA to deal with SDP for us, since dealing with SDP's manually is quite a tedious operation. With this approach, we can ask PJMEDIA to generate SDP for us (the pjmedia_endpt_create_sdp() function) and we can use the pjmedia_session_info_from_sdp() function to convert local and remote SDP's into the more manageable ​pjmedia_session_info structure.

Pros:

• much easier to work with than manipulating SDP's manually
• this approach is supported since it doesn't modify our codes/libraries

Cos:

• there is a bit code needed to register codecs into PJMEDIA.
• it still can only work with PJSIP (and not PJSUA-LIB)

Steps to use this approach:

1. Application needs to link with PJMEDIA library to get the SDP, the SDP generation/conversion and codec information parts of PJMEDIA.
2. Implement a codec factory for each codec that the third party media stack supports, to register the codec information into PJMEDIA. Note that only the codec factory needs to be implemented, since we will never create PJMEDIA's stream so no codec will ever gets instantiated.
3. The operations in ​pjmedia_codec_factory_op that need to be implemented are:
   • test_alloc()
   • default_attr()
   • enum_info()
4. You can use the sample codec implementations as a template to create your codec factories. Hints: you may use one codec factory to register multiple types of codecs.
5. Create a pjmedia_endpt instance with ​pjmedia_endpt_create(), and registers each of the codec factories into the codec manager (application can query the codec manager instance with ​pjmedia_endpt_get_codec_mgr() function).
6. The code to create a call or to handle media update event is similar to the generic pseudo-code explained earlier, except that application will replace call to ​pjmedia_session_create() with the corresponding function provided by the third party media stack to start the media.

---

Integrating Third Party Media Stack into PJMEDIA

The basic idea here is to integrate the third party media stack into PJMEDIA, and since the PJMEDIA API stays the same, we can integrate the resulting library with PJSUA-LIB. However, since this involves modifying our code/libraries, this approach is not supported.

We never tried this approach (never had the need!), so we can't quite explain the details on how to accomplish this. Also the details will depend on the capability that is offered by the third party media stack, so it varies. But the rough steps probably look something like these:

1. Since we no longer use PJMEDIA to control the media, we need to stop the media in PJSUA-LIB from running. Follow this article on PJSIP FAQ on how to manage our own media with PJSUA-LIB.
2. Re-implement pjmedia_session_create() function in PJMEDIA's session.c with your implementation (that uses the third party media stack). You may just return a dummy session here if you decide to handle media session creation in the application code (in on_stream_created() callback of PJSUA-LIB. Please see the FAQ article above for the details).
3. You'd also probably need to re-implement the UDP media transport. The media transport is needed because PJSUA-LIB will instantiate them to get their socket addresses (to be advertised in SDP). You still need to implement the UDP media transport even when your media stack handles the transport (because PJSUA-LIB still needs to retrieve the socket addresses); in this case, it needs to be re-implemented so that it's not actually binding to the specified socket address, or otherwise if your media stack tries to bind to the same address, it will fail.

With these installed, I think you can get PJSUA-LIB up and running. You will however lose all media functionality in PJSUA-LIB, since the media is not running in PJSUA-LIB, such as:

1. conference bridge
2. file playing recording
3. ICE
4. everything else related to media