wiki:3rd_Party_Media

Version 5 (modified by bennylp, 11 years ago) (diff)

--

Integrating Third Party Media Stack with PJSIP

Table of Contents

  1. Introduction
  2. SIP and Media Interaction
    1. Creating a Call/Invite? Session
    2. Starting the Media
    3. Subsequent Offer/Answer?
  3. Third Party Integration Strategy
    1. Completely Replace PJMEDIA with Third Party Media Stack
    2. Reusing PJMEDIA's Session Info
    3. Integrating Third Party Media Stack into PJMEDIA


Introduction

PJSIP consists of several libraries which mostly are independent from one another. The library architecture diagram below shows how libraries are interacting with each other.

PJSUA-LIB:

This is a library which integrates all other libraries into very high level API. Due to its nature, it is tightly coupled with all of the libraries.

SIP Libraries:

This includes PJSIP core library (pjsip-core), SIP user agent client library (pjsip-ua), and SIP presence and instant messaging library (pjsip-simple).

Media Stack:

The media stack is PJMEDIA and PJMEDIA-CODEC.

SDP components:

Although the SDP components (message representation, parsing, and SDP offer answer negotiation) is packaged with PJMEDIA, actually it is independent from PJMEDIA and it can be (and has been) used independently from PJMEDIA. This is because although media can be optional for some application (for example, a server), SDP and SDP negotiation are needed by the invite session, so it should be able to use these parts without having to use PJMEDIA.

In fact in Symbian build system, these SDP parts are packaged on separate static library (as pjsdp library). On GNU build system, both PJMEDIA and PJSDP libraries are built, although only PJMEDIA is linked with the application since unlike the Symbian build system, the PJMEDIA library here also contains PJSDP parts.


SIP and Media Interaction

Before we discuss how to integrate third party media library with PJSIP, we need to understand how PJSIP interacts with the media stack (PJMEDIA, in this case). This interaction/integration is done by either PJSUA-LIB or application, depending on whether the application uses PJSUA-LIB or PJSIP directly.

There are few places where PJSIP and PJMEDIA interact with each other:

  • During call/invite session creation: we need to specify local SDP when creating invite session.
  • After SDP negotiation: once SDP has been negotiated, we can start our media streaming.
  • Anytime thereafter when the invite session requires us to update our offer or answer.

These will be discussed in more detail in the following sections.

Creating a Call/Invite? Session

An invite session is created by application (or PJSUA-LIB) when we make outgoing call or when we receive incoming call. We need to specify our local SDP when creating the invite session, because SDP negotiation will be done by the invite session [*], and to do this it needs SDP from both local and remote endpoints. Normally application just need to ask PJMEDIA to create local SDP for us, based on capabilities that have been registered to it.

[*] To be precise, the negotiation actually is done by the SDP negotiator (part of PJMEDIA/PJSDP library) which will be invoked by the invite session. The decision to make SDP negotiation as integral part of invite session is to shield application from the complexity of SDP offer/answer session (for example, with late offer/answer, offer/answer quirks related to the use of PRACK, and ability to update session early with UPDATE), while maintaining the flexibility for the application to control when and what offer and answer to be given to remote.

Below is the pseudo-code for creating a call/invite session. The procedure is relatively similar for both caller and callee.

create_call()
{
    // Create the dialog
    dlg = pjsip_dlg_create_uac/uas();

    // Create media transport (pair of RTP/RTCP socket) for
    // transmitting/receiving the media from remote endpoint
    media_transport = pjmedia_transport_udp_create();

    // Get the socket address info of the media transport
    // These are needed to create a complete SDP descriptor
    media_sock_info = pjmedia_transport_get_info(media_transport);
   
    // Create local SDP to be given to invite session
    sdp = pjmedia_endpt_create_sdp(media_sock_info);

    // Create the invite session, giving it both the dialog
    // and local SDP to be negotiated
    inv = pjsip_inv_create_uac/uas(dlg, sdp);
}

To look what pjmedia_endpt_create_sdp() function is doing in detail, below is the pseudo-code of the function:

pjsip_endpt_create_sdp(media_sock_info)
{
  // Query list of codecs that we support from the codec manager
  codec_list[] = pjmedia_codec_mgr_enum_codecs();

  // Put each codec information in the SDP
  for each codec in codec_list {
    put codec info in SDP
  }

  // Put media socket address info in SDP c= and m= line 
  sdp->connection_line = media_sock_info->rtp.ip_address;
  sdp->m_line.port = media_sock_info->rtp.port_number;

  return sdp;
}

Starting the Media

As discussed above, SDP negotiation is done by the SDP negotiator object that is invoked internally by the invite session. The invite session user (application or PJSUA-LIB) will get notification whenever the SDP negotiation has been done, via invite session's on_media_update() callback.

The on_media_update() callback contains status parameter which tells application whether SDP negotiation has been successful (or not). We are normally only interested with the successful status to start our media.

Below is the pseudo-code to implement on_media_update() callback. You can see a working implementation of this function in pjsua_call_on_media_update() function of pjsua_call.c in PJSUA-LIB.

void on_media_update(inv, status)
{
   if (status != PJ_SUCCESS) {
      // Handle failed negotiation scenario
      ...
      return;
   }

   // Retrieve both local and remote active SDP.
   // Active SDP is the SDP that has been negotiated
   // by SDP negotiator.
   local_sdp = pjmedia_sdp_neg_get_active_local(inv->neg);
   remote_sdp = pjmedia_sdp_neg_get_active_remote(inv->neg);

   // Update media based on local and remote SDP
   update_media_channel(local_sdp, remote_sdp);
}

And below is the pseudo-code of update_media_channel() function. You can see a working implementation of this function in pjsua_media_channel_update() of pjsua_media.c in PJSUA-LIB.

The update_media_channel needs to handle two cases:

  • when media is activated
  • when media is deactivated (for example, on call hold)
update_media_channel(local_sdp, remote_sdp)
{
   // Create media session info from both local and remote SDP.
   // The media session contains everything that's needed to create our
   // media stream (codec settings, transport settings (local and remote
   // socket addresses), jitter buffer settings, etc.)
   media_sess_info = pjmedia_session_info_from_sdp(local_sdp, remote_sdp);

   // Handle inactive media
   if (media_sess_info->dir = NONE) {
      // Handle inactive media
      ..
      
      return;
   }

   // Customize media session info if wanted (for example, turn VAD/PLC 
   // on/off, change jitter buffer setting, set SSRC, etc).
   ..

   // Create the media session, giving it both the media session info
   // and media transport. We created the media transport earlier when
   // we create the invite session.
   m_session = pjmedia_session_create(media_sess_info, media_transport);

   // Extract audio stream from the media session
   stream_port = pjmedia_session_get_port(m_session, 0);

   // "Connect" the audio stream to sound device (or conference bridge,
   // or anything else).
   pjmedia_snd_port_connect(sound_device, stream_port);
}

Subsequent Offer/Answer?

The invite session provides additional callbacks relating to SDP offer/answer:

  • on_rx_offer(): this callback is called when the session has received a new offer from remote, and it needs a local SDP.
  • on_create_offer(): this callback is called when the session needs to generate a local offer (for example when it receives incoming re-INVITE without an offer).

For both of these two callbacks, application MUST provide a local SDP to the invite session. The process of getting local SDP is similar to the one when creating a call/invite session above.

For a working sample, please see pjsua_call_on_rx_offer() and pjsua_call_on_create_offer() functions in pjsua_call.c file of PJSUA-LIB.


Third Party Integration Strategy

We have two approaches on how to integrate your third party media stack with PJSIP:

  1. to completely remove PJMEDIA from application dependency
  2. reuse PJMEDIA's session info
  3. integrating third party stack into PJMEDIA

Please note than on all approaches, the SDP parts (messaging, parsing, and offer answer negotiation) cannot be excluded from application dependency since SDP is tightly coupled with the invite session (for the reasons explained above). Fortunately the SDP parts are independent from PJMEDIA thus can be taken out from PJMEDIA cleanly.

Both integration approaches above have their pros and cons which will be discussed below.

Completely Replace PJMEDIA with Third Party Media Stack

With this approach, basically we don't link/use PJMEDIA at all. Pros and cons of this approach are:

Pros:

  • the separation is cleaner thus may be easier to understand. Basically application MUST NOT use anything from PJMEDIA except <sdp.h> and <sdp_neg.h>
  • using this approach is supported, since it does not require modifications to our codes/libraries.

Cons:

  • This approach can only be used when application uses PJSIP directly (i.e. not PJSUA-LIB), thus it requires significant more efforts to accomplish tasks which otherwise can be done more easily with PJSUA-LIB. But on the positive note, PJSUA-LIB contains ready to use examples on how to do things so one may just need to copy/paste the code into application's code.
  • Since we are not allowed to use PJMEDIA's media session info, we need to get the media info directly from the SDP's, and this is tedious!

To use this approach, you need to do the following (more or less):

  1. Create a new library which consists of the following files from PJMEDIA directory (note that on GNU and Symbian build systems, a library containing these files has been provided and it's called pjsdp library):
    • errno.c
    • sdp.c
    • sdp_cmp.c
    • sdp_neg.c
  2. To create a local SDP for the invite session (either when creating the call or when the invite session needs updated offer or answer), you need to build the SDP manually. You cannot use pjmedia_endpt_create_sdp() function since this function is part of PJMEDIA.
  3. To create a media session, you need to manually inspect both local and remote SDP (see on_media_update() pseudo-code above) and create your media session based on the information in both SDP's.

Reusing PJMEDIA's Session Info

The idea of this approach is to allow PJMEDIA to deal with SDP for us, since dealing with SDP's manually is quite a tedious operation. With this approach, we can ask PJMEDIA to generate SDP for us (the pjmedia_endpt_create_sdp() function) based on the capability/codecs that are registered to it, and we can use the pjmedia_session_info_from_sdp() function to convert local and remote SDP's into the more manageable media session info structure.

Pros:

  • much easier to work with than manipulating SDP's manually
  • this approach is supported since it doesn't modify our codes/libraries

Cos:

  • there is a bit of glue code needed to make PJMEDIA works for us.
  • it still can only work with PJSIP (and not PJSUA-LIB)

Steps to use this approach:

  1. application needs to link with PJMEDIA library to get the SDP parts and the SDP generation/conversion to media session info functionality.
  2. implement a codec factory for each codec that the third party media stack supports. The codec factory is needed to register the codec into codec manager's codec list. Note that only the codec factory needs to be implemented (the pjmedia_codec_factory_op API); no need to implement the codec operation itself (the pjmedia_codec_op API), since we will never create PJMEDIA's stream so no ever codec will get instantiated.
  3. The operations that need to be implemented in pjmedia_codec_factory_op are:
    • test_alloc()
    • default_attr()
    • enum_info()
  4. create a pjmedia_endpt instance, and registers each of the codec factories into the codec manager (application can query the codec manager instance from the pjmedia_endpt).
  5. the code to create a call or to handle media update event is similar to the generic pseudo-code explained earlier, except that application will replace call to pjmedia_session_create() with the corresponding function provided by the third party media stack to start the media.

Integrating Third Party Media Stack into PJMEDIA

The basic idea here is to integrate the third party media stack into PJMEDIA, and since the PJMEDIA API stays the same, we can integrate the result with existing application or higher layer library, including PJSUA-LIB. However, since this involves modifying our code/libraries, this approach is not supported.

We never tried this approach (never had the need!), so we can't quite explain the details on how to accomplish this. Also the details will depend on the capability that is offered by the third party media stack, so it varies. But the rough steps probably look something like these:

  1. We probably no longer use PJMEDIA to control the media flow, and if this is the case, we'll need to stop the media from running. Follow this article on PJSIP FAQ on how to manage our own media with PJSUA-LIB.
  2. Re-implement pjmedia_session_create() function in PJMEDIA's session.c with your implementation (that uses the third party media stack). Also re-implement:
    • pjmedia_session_destroy(): to destroy the media session
    • pjmedia_session_get_port(): just return a dummy media port. This wouldn't matter since media flow is not running in PJSUA-LIB so the media port will never be used.
    • pjmedia_session_dial_dtmf(), pjmedia_session_check_dtmf(), pjmedia_session_get_dtmf(), and pjmedia_session_set_dtmf_callback()
  3. You'd also probably need to re-implement the UDP media transport. The media transport is needed to get its socket address info (to be advertised in SDP), but it needs to be re-implemented to not actually bind to the specified socket address (otherwise if your media stack tries to bind to the same address, it will fail).

With these installed, I think you can get PJSUA-LIB up and running. You will however lose all media functionality in PJSUA-LIB, since the media is not running in PJSUA-LIB, such as:

  1. conference bridge
  2. file playing recording
  3. ICE
  4. everything else related to media

Attachments (1)

Download all attachments as: .zip