wiki:3rd_Party_Media

Version 2 (modified by bennylp, 11 years ago) (diff)

--

Integrating Third Party Media Stack with PJSIP

Introduction

PJSIP consists of several libraries which mostly are independent from one another. The library architecture diagram below shows how libraries are interacting with each other.

SIP and Media Interaction

Before we discuss how to integrate third party media library with PJSIP, we need to understand how PJSIP interacts with the media stack (PJMEDIA, in this case). This interaction/integration is done by either PJSUA-LIB or application, depending on whether the application uses PJSUA-LIB or PJSIP directly.

There are few places where PJSIP and PJMEDIA interact with each other:

  • During call/invite session creation: we need to specify local SDP when creating invite session.
  • After SDP negotiation: once SDP has been negotiated, we can start our media streaming.
  • Anytime thereafter when the invite session requires us to update our offer or answer.

These will be discussed in more detail in the following sections.

Creating a Call/Invite? Session

An invite session is created by application (or PJSUA-LIB) when we make outgoing call or when we receive incoming call. We need to specify our local SDP when creating the invite session, because SDP negotiation will be done by the invite session [*], and to do this it needs SDP from both local and remote endpoints. Normally application just need to ask PJMEDIA to create local SDP for us, based on capabilities that have been registered to it.

[*] To be precise, the negotiation actually is done by the SDP negotiator (part of PJMEDIA/PJSDP library) which will be invoked by the invite session. The decision to make SDP negotiation as integral part of invite session is to shield application from the complexity of SDP offer/answer session (for example, with late offer/answer, offer/answer quirks related to the use of PRACK, and ability to update session early with UPDATE), while maintaining the flexibility for the application to control when and what offer and answer to be given to remote.

Below is the pseudo-code for creating a call/invite session. The procedure is relatively similar for both caller and callee.

create_call()
{
    // Create the dialog
    dlg = pjsip_dlg_create_uac/uas();

    // Create media transport (pair of RTP/RTCP socket) for
    // transmitting/receiving the media from remote endpoint
    media_transport = pjmedia_transport_udp_create();

    // Get the socket address info of the media transport
    // These are needed to create a complete SDP descriptor
    media_sock_info = pjmedia_transport_get_info(media_transport);
   
    // Create local SDP to be given to invite session
    sdp = pjmedia_endpt_create_sdp(media_sock_info);

    // Create the invite session, giving it both the dialog
    // and local SDP to be negotiated
    inv = pjsip_inv_create_uac/uas(dlg, sdp);
}

To look what pjmedia_endpt_create_sdp() function is doing in detail, below is the pseudo-code of the function:

pjsip_endpt_create_sdp(media_sock_info)
{
  // Query list of codecs that we support from the codec manager
  codec_list[] = pjmedia_codec_mgr_enum_codecs();

  // Put each codec information in the SDP
  for each codec in codec_list {
    put codec info in SDP
  }

  // Put media socket address info in SDP c= and m= line 
  sdp->connection_line = media_sock_info->rtp.ip_address;
  sdp->m_line.port = media_sock_info->rtp.port_number;

  return sdp;
}

Starting the Media

As discussed above, SDP negotiation is done by the SDP negotiator object that is invoked internally by the invite session. The invite session user (application or PJSUA-LIB) will get notification whenever the SDP negotiation has been done, via invite session's on_media_update() callback.

The on_media_update() callback contains status parameter which tells application whether SDP negotiation has been successful (or not). We are normally only interested with the successful status to start our media.

Below is the pseudo-code to implement on_media_update() callback. You can see a working implementation of this function in pjsua_call_on_media_update() function of pjsua_call.c.

!#c
void on_media_update(inv, status)
{
   if (status != PJ_SUCCESS) {
      // Handle failed negotiation scenario
      ...
      return;
   }

   // Retrieve both local and remote active SDP.
   // Active SDP is the SDP that has been negotiated
   // by SDP negotiator.
   local_sdp = pjmedia_sdp_neg_get_active_local(inv->neg);
   remote_sdp = pjmedia_sdp_neg_get_active_remote(inv->neg);

   // Update media based on local and remote SDP
   update_media_channel(local_sdp, remote_sdp);
}

And below is the pseudo-code of update_media_channel() function. You can see a working implementation of this function in pjsua_media_channel_update() of pjsua_media.c.

The update_media_channel needs to handle two cases:

  • when media is activated
  • when media is deactivated (for example, on call hold)
!#c
update_media_channel(local_sdp, remote_sdp)
{
   // Create media session info from both local and remote SDP.
   // The media session contains everything that's needed to create our
   // media stream (codec settings, transport settings (local and remote
   // socket addresses), jitter buffer settings, etc.)
   media_sess_info = pjmedia_session_info_from_sdp(local_sdp, remote_sdp);

   // Handle inactive media
   if (media_sess_info->dir = NONE) {
      // Handle inactive media
      ..
      
      return;
   }

   // Customize media session info if wanted (for example, turn VAD/PLC 
   // on/off, change jitter buffer setting, set SSRC, etc).
   ..

   // Create the media session, giving it both the media session info
   // and media transport. We created the media transport earlier when
   // we create the invite session.
   m_session = pjmedia_session_create(media_sess_info, media_transport);

   // Extract audio stream from the media session
   stream_port = pjmedia_session_get_port(m_session, 0);

   // "Connect" the audio stream to sound device (or conference bridge,
   // or anything else).
   pjmedia_snd_port_connect(sound_device, stream_port);
}

Attachments (1)

Download all attachments as: .zip