Changes between Version 6 and Version 7 of 3rd_Party_Media


Ignore:
Timestamp:
Mar 16, 2008 11:24:44 AM (11 years ago)
Author:
bennylp
Comment:

--

Legend:

Unmodified
Added
Removed
Modified
  • 3rd_Party_Media

    v6 v7  
    3131---- 
    3232 
    33 == SIP and Media Interaction == 
    34  
    35 Before we discuss how to integrate third party media library with PJSIP, we need to understand how PJSIP interacts with the media stack (PJMEDIA, in this case). This interaction/integration is done by either PJSUA-LIB or application, depending on whether the application uses PJSUA-LIB or PJSIP directly. 
    36  
    37 There are few places where PJSIP and PJMEDIA interact with each other: 
    38  * '''During call/invite session creation''': we need to specify local SDP when creating invite session. 
    39  * '''After SDP negotiation''': once SDP has been negotiated, we can start our media streaming. 
     33== SIP and Media Integration == 
     34 
     35Depending on which API abstraction layer being used by application (PJSUA-LIB or PJSIP/PJMEDIA API directly), integration between PJSIP and PJMEDIA happens in either PJSUA-LIB library or in application code. In this integration, there are few places where PJSIP and PJMEDIA interact with each other: 
     36 * During call/invite session creation -- we need to specify local SDP when creating invite session. 
     37 * After SDP negotiation -- once SDP has been negotiated, we can start our media streaming. 
    4038 * Anytime thereafter when the invite session requires us to update our offer or answer. 
    4139 
     
    4442=== Creating a Call/Invite Session === 
    4543 
    46 An invite session is created by application (or PJSUA-LIB) when we make outgoing call or when we receive incoming call. We need to specify our local SDP when creating the invite session, because SDP negotiation will be done by the invite session [*], and to do this it needs SDP from both local and remote endpoints. Normally application just need to ask PJMEDIA to create local SDP for us, based on capabilities that have been registered to it. 
    47  
    48   ''[*] To be precise, the negotiation actually is done by the SDP negotiator (part of PJMEDIA/PJSDP library) which will be invoked by the invite session. The decision to make SDP negotiation as integral part of invite session is to shield application from the complexity of SDP offer/answer session (for example, with late offer/answer, offer/answer quirks related to the use of PRACK, and ability to update session early with UPDATE), while maintaining the flexibility for the application to control when and what offer and answer to be given to remote.'' 
    49  
    50 Below is the pseudo-code for creating a call/invite session. The procedure is relatively similar for both caller and callee. 
     44An [http://www.pjsip.org/pjsip/docs/html/group__PJSIP__INV.htm invite session] is created by application (or PJSUA-LIB) when we make outgoing call or when we receive incoming call. We need to specify our local SDP when creating the invite session, because SDP negotiation will be done by the invite session [*], and to do this it needs SDP from both local and remote endpoints. 
     45 
     46  ''[*] To be precise, the negotiation actually is done by the [http://www.pjsip.org/pjmedia/docs/html/group__PJMEDIA__SDP__NEG.htm SDP negotiator] (part of PJMEDIA/PJSDP library) which will be invoked by the invite session. The motivation to make SDP negotiation as integral part of invite session is to shield application from the complexity of SDP offer/answer session (for example, with late offer/answer, offer/answer quirks related to the use of PRACK, and ability to update session early with UPDATE), while maintaining the flexibility for the application to control when and what offer and answer to be given to remote.'' 
     47 
     48Below is the pseudo-code for creating a call/invite session. The procedure is relatively similar for both caller and callee. For a working sample, please see [http://www.pjsip.org/pjsip/docs/html/page_pjsip_sample_simple_ua_c.htm '''simpleua''' sample] or PJSUA-LIB source code. 
    5149 
    5250{{{ 
     
    6260 
    6361    // Get the socket address info of the media transport 
    64     // These are needed to create a complete SDP descriptor 
     62    // The socket address will be put in SDP c= and m= lines. 
    6563    media_sock_info = pjmedia_transport_get_info(media_transport); 
    6664    
    67     // Create local SDP to be given to invite session 
     65    // The easiest way to create SDP is to ask the media endpoint 
     66    // to create an SDP describing local media capability for us. 
     67    // The function needs the media socket address which will be 
     68    // placed in SDP c= and m= line. 
    6869    sdp = pjmedia_endpt_create_sdp(media_sock_info); 
    6970 
    7071    // Create the invite session, giving it both the dialog 
    71     // and local SDP to be negotiated 
     72    // and local SDP to be negotiated. 
    7273    inv = pjsip_inv_create_uac/uas(dlg, sdp); 
    7374} 
    7475}}} 
    7576 
    76 To look what ''pjmedia_endpt_create_sdp()'' function is doing in detail, below is the pseudo-code of the function: 
     77It may be worthwhile to look what [http://www.pjsip.org/pjmedia/docs/html/group__PJMED__ENDPT.htm#ga49360a690c292dddd722b39c2f8c8bb pjmedia_endpt_create_sdp()] function is doing in detail, so below is the pseudo-code of the function: 
    7778 
    7879{{{ 
     
    8081pjsip_endpt_create_sdp(media_sock_info) 
    8182{ 
     83  // Create blank SDP structure 
     84  sdp = alloc(sizeof(pjmedia_sdp_session)); 
     85 
     86  // Fill in base SDP fields (such as t=, s=, c= lines) 
     87  .. 
     88 
    8289  // Query list of codecs that we support from the codec manager 
    8390  codec_list[] = pjmedia_codec_mgr_enum_codecs(); 
     
    96103}}} 
    97104 
     105For further reference, please see: 
     106 * [http://www.pjsip.org/pjsip/docs/html/group__PJSIP__INV.htm PJSIP Invite Session] object and its callbacks. 
     107 * [http://www.pjsip.org/pjsip/docs/html/group__PJSIP__DIALOG.htm PJSIP Dialog] object 
     108 * [http://www.pjsip.org/pjmedia/docs/html/group__PJMEDIA__SDP__NEG.htm SDP Negotiator] object 
     109 * [http://www.pjsip.org/pjmedia/docs/html/group__PJMEDIA__SDP.htm SDP representation and parsing] 
     110 
     111 
    98112 
    99113=== Starting the Media === 
    100114 
    101 As discussed above, SDP negotiation is done by the SDP negotiator object that is invoked internally by the invite session. The invite session user (application or PJSUA-LIB) will get notification whenever the SDP negotiation has been done, via invite session's '''on_media_update()''' callback. 
    102  
    103 The ''on_media_update()'' callback contains ''status'' parameter which tells application whether SDP negotiation has been successful (or not). We are normally only interested with the successful status to start our media. 
    104  
    105 Below is the pseudo-code to implement ''on_media_update()'' callback. You can see a working implementation of this function in ''pjsua_call_on_media_update()'' function of {{{pjsua_call.c}}} in PJSUA-LIB. 
     115As discussed above, SDP negotiation is done by the [http://www.pjsip.org/pjmedia/docs/html/group__PJMEDIA__SDP__NEG.htm SDP negotiator] object that is invoked internally by [http://www.pjsip.org/pjsip/docs/html/group__PJSIP__INV.htm the invite session]. The invite session user (application or PJSUA-LIB) will get notification whenever the SDP negotiation has been done, via invite session's '''on_media_update()''' callback. 
     116 
     117The [http://www.pjsip.org/pjsip/docs/html/structpjsip__inv__callback.htm#aafbe25fa41f5aa57dfe5a42bf025f2d on_media_update()] callback contains ''status'' parameter which tells application whether SDP negotiation has been successful (or not). We are normally only interested with the successful status to start our media. 
     118 
     119Below is the pseudo-code to implement ''on_media_update()'' callback. You can see a working implementation of this function in [http://www.pjsip.org/pjsip/docs/html/page_pjsip_sample_simple_ua_c.htm simpleua sample] or ''pjsua_call_on_media_update()'' function of {{{pjsua_call.c}}} in PJSUA-LIB. 
    106120 
    107121{{{ 
     
    110124{ 
    111125   if (status != PJ_SUCCESS) { 
    112       // Handle failed negotiation scenario 
     126      // Handle failed negotiation scenario.  
     127      // Normally we don't need to do anything here 
    113128      ... 
    114129      return; 
    115130   } 
    116131 
    117    // Retrieve both local and remote active SDP. 
     132   // Retrieve both local and remote active SDP's. 
    118133   // Active SDP is the SDP that has been negotiated 
    119    // by SDP negotiator. 
     134   // by SDP negotiator, and the content may differ from 
     135   // the initial content when the SDP's are given to 
     136   // the invite session (for example, codec order may be 
     137   // rearranged, and non-active codecs may be removed 
     138   // altogether). 
    120139   local_sdp = pjmedia_sdp_neg_get_active_local(inv->neg); 
    121140   remote_sdp = pjmedia_sdp_neg_get_active_remote(inv->neg); 
     
    128147And below is the pseudo-code of ''update_media_channel()'' function. You can see a working implementation of this function in ''pjsua_media_channel_update()'' of pjsua_media.c in PJSUA-LIB. 
    129148 
    130 The ''update_media_channel'' needs to handle two cases: 
    131  * when media is activated 
    132  * when media is deactivated (for example, on call hold) 
    133149 
    134150{{{ 
     
    151167 
    152168   // Customize media session info if wanted (for example, turn VAD/PLC  
    153    // on/off, change jitter buffer setting, set SSRC, etc). 
     169   // on/off, change jitter buffer setting, set RTP/RTCP SSRC, etc). 
    154170   .. 
    155171 
     
    168184}}} 
    169185 
     186For further reference, please see: 
     187 * [http://www.pjsip.org/pjmedia/docs/html/group__PJMED__SES.htm Media session] object 
     188 * [http://www.pjsip.org/pjmedia/docs/html/group__PJMED__SND__PORT.htm Sound device port] object 
     189 * [http://www.pjsip.org/pjmedia/docs/html/group__PJMEDIA__PORT__CONCEPT.htm Media port framework] in PJMEDIA 
     190 
     191 
    170192=== Subsequent Offer/Answer === 
    171193 
    172 The invite session provides additional callbacks relating to SDP offer/answer: 
     194The invite session provides additional callbacks related to SDP offer/answer negotiation: 
    173195 * '''on_rx_offer()''': this callback is called when the session has received a new offer from remote, and it needs a local SDP. 
    174196 * '''on_create_offer()''': this callback is called when the session needs to generate a local offer (for example when it receives incoming re-INVITE without an offer). 
    175197 
    176 For both of these two callbacks, application MUST provide a local SDP to the invite session. The process of getting local SDP is similar to the one when creating a call/invite session above. 
    177  
    178 For a working sample, please see ''pjsua_call_on_rx_offer()'' and ''pjsua_call_on_create_offer()'' functions in {{{pjsua_call.c}}} file of PJSUA-LIB. 
     198Application MUST provide a local SDP to the invite session in both callbacks. The process of getting local SDP is similar to the one when creating a call/invite session above. For a working sample, please see ''pjsua_call_on_rx_offer()'' and ''pjsua_call_on_create_offer()'' functions in {{{pjsua_call.c}}} file of PJSUA-LIB. 
     199 
     200For further reference, please see: 
     201 * [http://www.pjsip.org/pjsip/docs/html/structpjsip__inv__callback.htm Invite Session callback] 
     202 
    179203 
    180204---- 
     
    182206== Third Party Integration Strategy == 
    183207 
    184 We have two approaches on how to integrate your third party media stack with PJSIP: 
    185  1. to completely remove PJMEDIA from application dependency 
    186  1. reuse PJMEDIA's session info 
     208We have three approaches on how to integrate your third party media stack with PJSIP: 
     209 1. to only use SDP components of PJMEDIA and nothing else (the SDP messaging, parsing, and offer answer negotiation are still needed since they are tightly coupled with the invite session for the reasons explained above). 
     210 1. above, plus to reuse PJMEDIA's session info generation to avoid working with SDP directly. 
    187211 1. integrating third party stack into PJMEDIA 
    188212 
    189 Please note than on all approaches, the SDP parts (messaging, parsing, and offer answer negotiation) cannot be excluded from application dependency since SDP is tightly coupled with the invite session (for the reasons explained above). Fortunately the SDP parts are independent from PJMEDIA thus can be taken out from PJMEDIA cleanly. 
    190  
    191 Both integration approaches above have their pros and cons which will be discussed below. 
     213Each integration approach have their pros and cons which will be discussed below. 
    192214 
    193215 
    194216=== Completely Replace PJMEDIA with Third Party Media Stack === 
    195217 
    196 With this approach, basically we don't link/use PJMEDIA at all. Pros and cons of this approach are: 
     218With this approach, basically we don't link/use PJMEDIA at all, except the SDP components of course. Pros and cons of this approach are: 
    197219 
    198220Pros: 
     
    202224Cons: 
    203225 * This approach can only be used when application uses PJSIP directly (i.e. not PJSUA-LIB), thus it requires significant more efforts to accomplish tasks which otherwise can be done more easily with PJSUA-LIB. But on the positive note, PJSUA-LIB contains ready to use examples on how to do things so one may just need to copy/paste the code into application's code. 
    204  * Since we are not allowed to use PJMEDIA's media session info, we need to get the media info directly from the SDP's, and this is tedious! 
     226 * Since we are not allowed to use PJMEDIA's media session info, we need to manually inspect the SDP fields on both local and remote SDP's to determine the parameters for the media session. This could be a tedious operation. 
    205227 
    206228To use this approach, you need to do the following (more or less): 
    207  1. Create a new library which consists of the following files from PJMEDIA directory (note that on GNU and Symbian build systems, a library containing these files has been provided and it's called {{{pjsdp}}} library): 
    208     - errno.c 
    209     - sdp.c 
    210     - sdp_cmp.c 
    211     - sdp_neg.c 
    212  1. To create a local SDP for the invite session (either when creating the call or when the invite session needs updated offer or answer), you need to build the SDP manually. You cannot use {{{pjmedia_endpt_create_sdp()}}} function since this function is part of PJMEDIA. 
     229 1. Link with the appropriate library: 
     230    - On Symbian or GNU build system, link the application with PJSDP library and exclude PJMEDIA from the link process. 
     231    - On other build system, you may link with PJMEDIA to get the SDP components linked with application, but do not use anything else. 
     232    - Alternatively you may create a new library which consists of the following files from PJMEDIA directory: errno.c, sdp.c, sdp_cmp.c, and sdp_neg.c 
     233 1. You need to build the SDP manually for the invite session (either when creating the call or when the invite session needs updated offer or answer). You cannot use {{{pjmedia_endpt_create_sdp()}}} function since this function is part of PJMEDIA. 
    213234 1. To create a media session, you need to manually inspect both local and remote SDP (see ''on_media_update()'' pseudo-code above) and create your media session based on the information in both SDP's. 
    214235 
     
    216237=== Reusing PJMEDIA's Session Info === 
    217238 
    218 The idea of this approach is to allow PJMEDIA to deal with SDP for us, since dealing with SDP's manually is quite a tedious operation. With this approach, we can ask PJMEDIA to generate SDP for us (the ''pjmedia_endpt_create_sdp()'' function) based on the capability/codecs that are registered to it, and we can use the ''pjmedia_session_info_from_sdp()'' function to convert local and remote SDP's into the more manageable media session info structure. 
     239The idea of this approach is to allow PJMEDIA to deal with SDP for us, since dealing with SDP's manually is quite a tedious operation. With this approach, we can ask PJMEDIA to generate SDP for us (the ''pjmedia_endpt_create_sdp()'' function) and we can use the ''pjmedia_session_info_from_sdp()'' function to convert local and remote SDP's into the more manageable [http://www.pjsip.org/pjmedia/docs/html/structpjmedia__session__info.htm pjmedia_session_info] structure. 
    219240 
    220241Pros: 
     
    223244 
    224245Cos: 
    225  * there is a bit of glue code needed to make PJMEDIA works for us. 
     246 * there is a bit code needed to register codecs into PJMEDIA. 
    226247 * it still can only work with PJSIP (and not PJSUA-LIB) 
    227248 
    228249 
    229250Steps to use this approach: 
    230  1. application needs to link with PJMEDIA library to get the SDP parts and the SDP generation/conversion to media session info functionality. 
    231  1. implement a codec factory for each codec that the third party media stack supports. The codec factory is needed to register the codec into codec manager's codec list. Note that only the codec factory needs to be implemented (the ''pjmedia_codec_factory_op'' API); no need to implement the codec operation itself (the ''pjmedia_codec_op'' API), since we will never create PJMEDIA's stream so no ever codec will get instantiated. 
    232  1. The operations that need to be implemented in ''pjmedia_codec_factory_op'' are: 
     251 1. Application needs to link with PJMEDIA library to get the SDP, the SDP generation/conversion and codec information parts of PJMEDIA. 
     252 1. Implement a codec factory for each codec that the third party media stack supports, to register the codec information into PJMEDIA. Note that only the codec factory needs to be implemented, since we will never create PJMEDIA's stream so no codec will ever gets instantiated. 
     253 1. The operations in ''pjmedia_codec_factory_op'' that need to be implemented are: 
    233254  - test_alloc() 
    234255  - default_attr() 
    235256  - enum_info() 
    236  1. create a ''pjmedia_endpt'' instance, and registers each of the codec factories into the codec manager (application can query the codec manager instance from the ''pjmedia_endpt''). 
    237  1. the code to create a call or to handle media update event is similar to the generic pseudo-code explained earlier, except that application will replace call to ''pjmedia_session_create()'' with the corresponding function provided by the third party media stack to start the media. 
     257 1. You can use the sample codec implementations as a template to create your codec factories. ''Hints: you may use one codec factory to register multiple types of codecs''. 
     258 1. Create a ''pjmedia_endpt'' instance, and registers each of the codec factories into the codec manager (application can query the codec manager instance with [http://www.pjsip.org/pjmedia/docs/html/group__PJMED__ENDPT.htm#ge058aeedcfad28aa6293308c65429760 pjmedia_endpt_get_codec_mgr()] function). 
     259 1. The code to create a call or to handle media update event is similar to the generic pseudo-code explained earlier, except that application will replace call to [http://www.pjsip.org/pjmedia/docs/html/group__PJMED__SES.htm#g6bb6bcc4721ca9f357b5e3bb750378c6 pjmedia_session_create()] with the corresponding function provided by the third party media stack to start the media. 
    238260 
    239261 
    240262=== Integrating Third Party Media Stack into PJMEDIA === 
    241263 
    242 The basic idea here is to integrate the third party media stack into PJMEDIA, and since the PJMEDIA API stays the same, we can integrate the result with existing application or higher layer library, including PJSUA-LIB. However, since this involves modifying our code/libraries, this approach is not supported. 
     264The basic idea here is to integrate the third party media stack into PJMEDIA, and since the PJMEDIA API stays the same, we can integrate the resulting library with PJSUA-LIB. However, since this involves modifying our code/libraries, this approach is not supported. 
    243265 
    244266We never tried this approach (never had the need!), so we can't quite explain the details on how to accomplish this. Also the details will depend on the capability that is offered by the third party media stack, so it varies. But the rough steps probably look something like these: 
    245  1. We probably no longer use PJMEDIA to control the media flow, and if this is the case, we'll need to stop the media from running. Follow [wiki:FAQ#pjsua-lib-perf this article on PJSIP FAQ] on how to manage our own media with PJSUA-LIB. 
    246  1. Re-implement ''pjmedia_session_create()'' function in PJMEDIA's {{{session.c}}} with your implementation (that uses the third party media stack). Also re-implement: 
    247   - pjmedia_session_destroy(): to destroy the media session 
    248   - pjmedia_session_get_port(): just return a dummy media port. This wouldn't matter since media flow is not running in PJSUA-LIB so the media port will never be used. 
    249   - pjmedia_session_dial_dtmf(), pjmedia_session_check_dtmf(), pjmedia_session_get_dtmf(), and pjmedia_session_set_dtmf_callback() 
    250  1. You'd also probably need to re-implement the UDP media transport. The media transport is needed to get its socket address info (to be advertised in SDP), but it needs to be re-implemented to '''not''' actually bind to the specified socket address (otherwise if your media stack tries to bind to the same address, it will fail). 
     267 1. Since we no longer use PJMEDIA to control the media, we need to stop the media in PJSUA-LIB from running. Follow [wiki:FAQ#pjsua-lib-perf this article on PJSIP FAQ] on how to manage our own media with PJSUA-LIB. 
     268 1. Re-implement ''pjmedia_session_create()'' function in PJMEDIA's {{{session.c}}} with your implementation (that uses the third party media stack). You may just return a dummy session here if you decide to handle media session creation in the application code (in ''on_stream_created()'' callback of PJSUA-LIB. Please see the FAQ article above for the details). 
     269 1. You'd also probably need to re-implement the UDP media transport. The media transport is needed because PJSUA-LIB will instantiate them to get their  socket addresses (to be advertised in SDP). You still need to implement the UDP media transport even when your media stack handles the transport (because PJSUA-LIB still needs to retrieve the socket addresses); in this case, it needs to be re-implemented so that it's '''not''' actually binding to the specified socket address, or otherwise if your media stack tries to bind to the same address, it will fail. 
    251270 
    252271With these installed, I think you can get PJSUA-LIB up and running. You will however lose '''all''' media functionality in PJSUA-LIB, since the media is not running in PJSUA-LIB, such as: 
     
    255274 1. ICE 
    256275 1. everything else related to media 
     276