Custom Query (2195 matches)

Passing an incorrect frame size to opus_decode() can cause decode error or recovery failure (log: "Recover failed!").

Scenario:

1. An account is registered using a TCP/TLS transport, as allow_via_rewrite is set, account's Via is rewritten with public IP.
2. The TCP/TLS transport gets disconnected.
3. INVITE for the account is sent using a new TCP/TLS transport, Via header for this INVITE is local IP address (of the new transport), this is as expected because no re-REGISTER has been sent and the account's Via address is no longer valid.
4. CANCEL is sent, but using current account's Via address while it should use the same Via header as the original INVITE to be cancelled, so this CANCEL is rejected with status code 481 (transaction does not exist).

See also #1996.

Thanks Marcus Froeschl for the report.

Support C# binding using SWIG. The resulting C# binding can then be used for C# apps, as well as for Xamarin projects.

Requirements:

1. Install swig-csharp
2. (For C#/Xamarin development): Download Visual Studio from ​xamarin.com.

How to create C# binding:

1. Go to pjsip-apps/src/swig.
2. Run make.

To create Xamarin sample app:

1. From Visual Studio, create new solution "Forms". Name it pjsua2xamarin in directory pjsip-apps/src/swig/csharp (we will refer this directory as [csharp_dir]).
2. In the multiplatform section (pjsua2xamarin.pjsua2xamarin):
   • Add pjsua2 folder ([csharp_dir]/pjsua2xamarin/pjsua2xamarin/pjsua2).
   • Add sample.cs file ([csharp_dir]/pjsua2xamarin/pjsua2xamarin/sample.cs).
3. Add code to run sample (such as in pjsua2xamarin.Droid MainActivity.cs or pjsua2xamarin.iOS AppDelegate.cs):

      pjsua2xamarin.sample test = new sample();
      test.test1();
   

4. Add PJSIP libraries.
   • For Android: Add lib folder ([csharp_dir]/Droid/lib) and change the Build Action of libpjsua2.so to Android Native Library. For more details, please refer to the ​official doc.
   • For iOS: Add the fat static library file (each architecture's resulting libpjsua2.a is located in [csharp_dir]/iOS/lib/[arch]). For more info, please refer to the ​official doc.
5. Set application permissions.
   See our sample apps (ipjsua for iOS and pjsua2 sample for Android) to see the typical basic permissions required.

Issues and solutions:

• The type or namespace name 'HandleRef' does not exist.
  SWIG requires .NET 2 or later by default and uses HandleRef. Make sure you are using the supported .NET framework version.
  ​https://github.com/swig/swig/issues/423
  ​https://github.com/swig/swig/issues/455
• TypeInitializationException, dll not found, unable to find pjsua2.
  This issue is likely caused by unsuccessful addition of the PJSIP libraries (see step 4 above), invalid path, or incorrect architecture of the libraries.
• Crashes when calling Endpoint.libInit(), or during initialization.
  This is likely caused by unauthorized permission (see step 5 above). App must list the required permissions and in some cases, specifically request for the permissions to the user, and user must grant those permissions.

In line with our roadmap for WebRTC interoperability, as mentioned in this draft of ​Web Real-Time Communication (WebRTC) document:

To reduce these costs and session set-up times,
   implementations are REQUIRED to support multiplexing RTP data packets
   and RTCP control packets on a single transport-layer flow.

this ticket will add support for RTP and RTCP multiplexing, in accordance to ​RFC 5761.

Currently PJMEDIA will only generate and answer SDP with one telephone-event format, i.e: telephone-event/8000. But in the stream session, it will send DTMF using timestamp correctly according to the audio codec clock rate.

This ticket will update the PJMEDIA behavior in SDP offer/answer:

• in generatig SDP offer, it will generate one or more telephone-event formats, one for each clock rate, in accordance with the clock rates of the offered audio codecs.
• in generating SDP answer, it will select telephone-event clock rate based on the clock rate of the selected audio codec.

From the ​RFC 7587:

The Opus encoder can output encoded frames representing 2.5, 5, 10,
   20, 40, or 60 ms of speech or audio data.  Further, an arbitrary
   number of frames can be combined into a packet, up to a maximum
   packet duration representing 120 ms of speech or audio data.

For example, for a packet duration of 40ms, we can receive 1 frame of 40ms, 2x20ms, 4x10ms, etc. However, currently pjmedia only expects constant frame length to be specified during stream creation.

Thank you to Marcus Froeschl for the suggestion and patch testing.

Ticket #1107 and #1225, described that iOS will reset UDP socket when app goes to background. The library will then try to recreate the socket by calling replace_udp_sock().

However since iOS11, we see some cases that the method fail and lead to unusable UDP transport or even worst, an unresponsive library state.

Log:

ioq_select !Attempting to replace UDP socket 5 
ioq_select  Error replacing socket: Invalid argument
udp0x127d27e00  Warning: pj_ioqueue_recvfrom: [err 120009] Bad file descriptor

Steps to reproduce:

• run ipjsua
• register to a registrar using UDP
• lock-unlock the phone repeatedly

This ticket will retry the recreate socket/replace_udp_sock() if it fail and add a fallback mechanism (restart transport) from the app callback.

Log:

ioq_select !Attempting to replace UDP socket 5
ioq_select  Error get peer name 120022
ioq_select  Error set qos param 120022
ioq_select  Retry to replace UDP socket 5
ioq_select !Error get peer name 120022
ioq_select  UDP has been replaced successfully!

A place for any bug fixes or enhancements related to DTLS-SRTP.

Currently when UDP media transport is created and attached, read operation is started immediately. Unfortunately, when remote starts sending RTP packet in the beginning of a call (especially when local is the SDP offerer), local stream may not be ready yet, so some initial inbound RTP packets will be read by the transport but then simply discarded as no stream is attached to the transport yet.

This ticket will change the behavior so read operation is started when pjmedia_transport_media_start() is called. So any incoming packet will be buffered by OS until read operation is started. This should be able to reduce missing video keyframe packets (or generally any initial RTP packets) as pjmedia_transport_media_start(), which starts read operation, is called after stream is instantiated.

Add support for SSRC attribute (​RFC 5576), and CNAME (​RFC 7022) in the SDP.

This ticket is created with the aim to support WebRTC interoperability according to this ​draft document:

Implementations are REQUIRED to support signalled RTP synchronisation
   source (SSRC) identifiers.

When a sender sends packets intensively to a receiver that has packet processing rate lower than the sending rate, the packet will be buffered by PJLIB SSL socket and after sometime the sender will get PJ_ENOMEM error and eventually drop the connection while there are still some pending outgoing packets in PJLIB SSL socket. Application cannot trace which packets have been sent and which packets are still pending (buffered in SSL socket).

After investigation, when sending buffer is full, any sending operation will still write to OpenSSL BIO while it should not, it should simply return error, e.g: PJ_ENOMEM. After sometime, it somehow cause OpenSSL to initiate SSL renegotiation and when renegotiation fails (e.g: due to timeout), SSL connection will be dropped.

Thanks Peter Koletzki for the report.

Currently SRTP crypto and keying method is configurable via callback, i.e: on_create_media_transport_srtp() in PJSUA or onCreateMediaTransportSrtp() in PJSUA2 (only crypto). And after some review and internal discussions, we decided to deprecate the callback and move the settings to account config, here are some reasons:

• SRTP setting type used in callback param is pjmedia_srtp_setting, which has some fields that are PJMEDIA specifics, e.g: close_member_tp, cb, user_data. So this ticket will also create a new SRTP settings specifically for PJSUA and PJSUA2.
• media transport settings are usually configurable via account setting (instead of PJSUA/PJSUA2 callback), so it'd better to use the same pattern for SRTP.

Here are sample codes to enable DTLS-SRTP in outgoing SDP offer:

• using PJSUA:

   acc_cfg.srtp_opt.keying_count = 2;
   acc_cfg.srtp_opt.keying[0] = PJMEDIA_SRTP_KEYING_DTLS_SRTP;
   acc_cfg.srtp_opt.keying[1] = PJMEDIA_SRTP_KEYING_SDES; // optional, as a fallback for handling incoming call using SRTP with SDES
  

• using PJSUA2:

  acc_cfg.mediaConfig.srtpOpt.keyings.push_back(PJMEDIA_SRTP_KEYING_DTLS_SRTP);
  acc_cfg.mediaConfig.srtpOpt.keyings.push_back(PJMEDIA_SRTP_KEYING_SDES); // optional, as a fallback for handling incoming call using SRTP with SDES
  

The PJLIB mutex functions all check for NULL mutexes but pj_atomic_destroy() is not setting its mutex to NULL after it destroyed it so any attempts to use the atomic again would cause a crash. This ticket does not address why an attempt was made to use the atomic again but it does prevent the crash.

Thank you George Joseph for the patch.

Reported that there have been cases that when the transaction timer callback is called when the transaction is already destroyed. This causes a crash. We now check the transaction state and return if the transaction is already destroyed.

Thank you George Joseph for the report and the patch.

In YUV format, if the buffer is all zero, it will be displayed as green (as compared to RGBA format, where it will be black). Unfortunately currently buffer is initialized with zero. As most video codecs uses I420 format, perhaps it is better to have special initialization for I420 so the screen will be black.

The allocation and initialization of frame buffer is done in pjmedia/src/pjmedia/vid_port.c, in function pjmedia_vid_port_create().

Failed STUN resolution when starting the library might lead to a crash caused by double free.

Scenario:

1. Set stun_ignore_failure to PJ_FALSE.
2. Turn networking OFF (Linux nework connections menu)
3. Start the application
4. pjsua_core.c !.STUN resolution failed: gethostbyname() has returned error (PJ_ERESOLVE).
5. pjsua_core.c .Error resolving STUN server: gethostbyname() has returned error (PJ_ERESOLVE) [status=70018]
6. double free, crash

Internal flow:

• pjsua_init() -> resolve_stun_server() : fail -> schedule timer to call destroy_stun_resolve_cb()
• exit pjsua_init() -> destroy_stun_resolve() -> shedule new timer to destroy STUN session. The previous timer entry hasn't been processed by worker thread
• double free and crash in busy_sleep()

Thank you to Denis Poltorak for the report.

Currently, running make in pjsip-apps/src/swig will create SWIG bindings for Java, Python, and the new addition C#, and may generate errors if one of them is not installed. It's better to allow user to specify an option such as make java to build for Java binding only. The list of options should be: java, python2, python3, csharp.

If possible, also detect the Python version used as well (for example, Python 3.6 or 2.7)

In AMR (narrow band), SID frames are of type 8. However in AMR-WB, SID frames are of type 9 (see 3GPP TS 26.201 Table 1a). This is also in line with ​RFC 4867: " ... or SID_BAD if the FT of the frame is 8 for AMR or 9 for AMR-WB... "

Currently pjsua always create media transports (UDP/ICE, with SRTP as an adapter). It may be desirable for some apps (such as the ones which use 3rd party media) to prevent this media tp creation. In this case, pjmedia_transport_loop can be used as an alternative.

According to the doc of pjsua_transport_config:

    /**
     * UDP port number to bind locally. This setting MUST be specified
     * even when default port is desired. If the value is zero, the
     * transport will be bound to any available port, and application
     * can query the port by querying the transport info.
     */
    unsigned		port;

However, currently setting the port to zero will result to it being assigned the default port number instead (which is set to 4000).

When receiving SIP message within/creating dialog, PJSIP should rewrite the Contact (and first Route if it is an initial INVITE transaction) if it is using IPv4 address, as otherwise, PJSIP will send a new request (within dialog) to that IPv4 address.

Thanks Kyle Kurz for the report.

In PJSIP stream, DTMF duration/timestamp is currently calculated using the actual clock rate, while it should be using the signalled rate. This will affect codecs such as Opus or G722.

Thanks to Marcus Froeschl for the report.

Currently the setting PJMEDIA_DTMF_DURATION specifies the duration in timestamp units. Thus, if using codecs with different clock rates, the time duration will vary. It may be desirable for some apps to be able to configure it in real time unit, i.e. in milliseconds, thus we introduce a new setting PJMEDIA_DTMF_DURATION_MSEC.

In OpenSSL 1.1.0 the ephemeral ECDH is already initialized in automatic mode, so there is really no need to do anything explicit about it.

=== begin citation ===
*) SSL_{CTX_}set_ecdh_auto() has been removed and ECDH is support is
     always enabled now.  If you want to disable the support you should
     exclude it using the list of supported ciphers. This also means that
the
     "-no_ecdhe" option has been removed from s_server.
https://www.openssl.org/news/changelog.html#x10
=== end citation ===

The code in ssl_sock_ossl.c falls to branch initializing only prime256v1 (aka secp256r1) elliptic curve in the context, after the call SSL_CTX_ctrl(ctx,94,1,NULL) is unsuccessful with OpenSSL 1.1.0x. When using server certificate with EC key based on any other curve, the listener fails TLS negotiation with misleading alert "no shared cipher", because the context's curve set applies to both EECDH and ECDSA. (Certificates with RSA keys work well.) Also, the EECDH itself is limited to use the only (from today's perspective the weakest acceptable) curve for key negotiation.

Thanks to Tzafrir Cohen for the patch.

In the doc of the API pjmedia_conf_connect_port(pjmedia_conf *conf, unsigned src_slot, unsigned sink_slot, int level):

@param level         This argument is reserved for future improvements
                     where it is possible to adjust the level of signal
                     transmitted in a specific connection. For now,
                     this argument MUST be zero.

This ticket will implement that improvement.

Thanks to Michael Scheiffler for the original patch and Thibault Groisil for subsequent improvements and fix.

On iOS when app goes into the background, the VideoToolbox? session will be invalid, thus it will generate continuous error. Then, when it returns to the foreground, this session will need to be reset in order for the encoding/decoding process to resume normally, otherwise the errors will continue and the video will freeze.

Thanks to Narayanan Kannan for the report.

The same issue as in ticket #1464 happens again.

The ticket doesn't entirely fix the problem, because PJSUA_LOCK() can be called by the upper functions instead, such as:

 	pjsua_call_on_state_changed(pjsip_inv_session * inv, pjsip_event * e) Line 4016	C
 	inv_set_state(pjsip_inv_session * inv, pjsip_inv_state state, pjsip_event * e) Line 317	C
 	inv_on_state_null(pjsip_inv_session * inv, pjsip_event * e) Line 3941	C
 	mod_inv_on_tsx_state(pjsip_transaction * tsx, pjsip_event * e) Line 717	C
 	pjsip_dlg_on_tsx_state(pjsip_dialog * dlg, pjsip_transaction * tsx, pjsip_event * e) Line 2069	C
 	mod_ua_on_tsx_state(pjsip_transaction * tsx, pjsip_event * e) Line 178	C
 	tsx_set_state(pjsip_transaction * tsx, pjsip_tsx_state_e state, pjsip_event_id_e event_src_type, void * event_src, int flag) Line 1268	C
 	tsx_on_state_null(pjsip_transaction * tsx, pjsip_event * event) Line 2483	C
 	pjsip_tsx_send_msg(pjsip_transaction * tsx, pjsip_tx_data * tdata) Line 1790	C
 	pjsip_dlg_send_request(pjsip_dialog * dlg, pjsip_tx_data * tdata, int mod_data_id, void * mod_data) Line 1288	C
 	pjsip_inv_send_msg(pjsip_inv_session * inv, pjsip_tx_data * tdata) Line 3282	C
 	on_make_call_med_tp_complete(int call_id, const pjsua_med_tp_state_info * info) Line 518	C
 	pjsua_call_make_call(int acc_id, const pj_str_t * dest_uri, const pjsua_call_setting * opt, void * user_data, const pjsua_msg_data * msg_data, int * p_call_id) Line 919	C

In the above stack trace, both pjsua_call_make_call() and on_make_call_med_tp_complete() both call PJSUA_LOCK().

The fix in this current ticket will improve the one in #1464 by ensuring that PJSUA_LOCK is not held before calling the callback. Note that if deadlock issue still persists, we may need to consider using group lock or chain the locks with pj_grp_lock_chain_lock()).

Thanks to Marcus Froeschl again for the report.

When testing pjsip on Raspberry Pi, it is reported that some weird sound issue happen if call lasted longer than ~30s. When checked with address sanitizer tools, the output is like this:

==3210==ERROR: AddressSanitizer: memcpy-param-overlap: memory ranges [0x6f5fd020,0x6f5fd1cc) and [0x6f5fd0c0, 0x6f5fd26c) overlap
    #0 0x59e37 in __interceptor_memcpy.part.36 (/home/pi/projects/sip/pjsip/simple_pjsua/simple_pjsua+0x59e37)
    #1 0x4dc79f in iLBC_encode ../../ilbc/iLBC_encode.c:311
    #2 0x2c7693 in ilbc_codec_encode ../src/pjmedia-codec/ilbc.c:754
    #3 0x3316f7 in pjmedia_codec_encode ../include/pjmedia/codec.h:1069

The issue is that iLBC sometimes uses memcpy() even when the source and the target storage overlap. This causes undefined behaviour and memmove() should be used in such cases (instead of memcpy()).

Similar issue was reported at ​https://issues.asterisk.org/jira/browse/ASTERISK-20231.

Thanks to Christian Hoff for the report and patch.

Scenario:

1. Thread 1 deletes SipAccount instance, which derived from PJSUA2 Account. In SipAccount destructor, some SipAccount member objects have been destroyed.
2. Thread 2 invokes Account callback onRegState() (e.g: from registration refresh), it tries to access SipAccount member objects, as some of them have been destroyed, crash occurs.

Some related facts:

1. Account destructor and onRegState() callback are mutual exclusive, because in PJSUA level, they are protected with PJSUA lock. But SipAccount destructor and onRegState() callback are not mutual exclusive.
2. Once pjsua_acc_del() in Account destructor is executed, onRegState() should never be invoked (for unregistration completion). Unfortunately, in derived class destruction, parent/Account destructor is called last.

The proposed solution is to introduce new Account method, i.e: Account::shutdown(), that internally will invoke PJSUA pjsua_acc_del(), so derived class could call this method first in its destructor to avoid invocation of onRegState() when it is being destroyed. Or alternatively, application can manually call Account::shutdown() before deleting the derived class instance.

Thanks Thomas Hackl for the report.

Currently stream buffer is calculated from specified codec maximum bitrate with 200 ms frame duration. Unfortunately, for some codecs that supports variable bitrate (VBR) and variable frame length such as Opus, it can be too small as the frame size may become very large exceeding the precalculated stream buffer size, which may lead to buffer overflow issues. Thanks Marcus Froeschl for the report and the analysis.

When there is no pjsua account available, and incoming call is received, assertion is raised:

pjsua_acc_find_for_incoming: assertion "pjsua_var.acc_cnt!=0" failed

In this case, assertion should be avoided since it is an uncontrollable event.

Thanks to Stanley Knapczyk for the report.

Scenario:

1. SIP server is configured with both Session Expires (SE) and Min-SE are set to 3600.
2. A & B clients are configured with SE & Min-SE set to default values (i.e: SE = 1800, Min-SE = 90).
3. A calls B, B receives INVITE with only Min-SE header and its value is 3600.
4. B answer the call without SE & Min-SE headers (session timer inactive).
5. B sends re-INVITE/UPDATE/call-hold with SE & Min-SE headers (attempt to activate session timer), unfortunately it uses SE=1800 while previously server has signaled that its Min-SE is 3600.
6. B receives 422 response as expected, and crashes.

After investigation, there seem to be a couple of bugs in the library:

1. When callee receives Min-SE header only, it doesn't update local SE to that Min-SE, so in any future outgoing request, the callee will use its original SE (which may be lower than caller's Min-SE and trigger 422 response).
2. The library wrongly assumes that 422 response can only occur in initial INVITE, while in the reported scenario it occurs in subsequent INVITE for call hold (due to bug #1 above), this is the main cause of the crash.

Thanks Shilpi Gupta for the report.

Sample exception message:

Swig::DirectorException: No enum class org.pjsip.pjsua2.pjsip_tsx_state_e
                         with value 377750600

SWIG requires C++ enum data to have a valid value so it can be correctly mapped to Java enum object. Unfortunately some PJSUA2 C++ objects do not initialize basic type data member (including enum) in its constructor and such uninitialized data member may have an invalid value, so when SWIG needs to map an invalid C++ enum value to Java, exception occurs.

Kazuhiko Yoneda for the report and the patch.

Currently libyuv converter backend does not support YV12/NV21 and when YV12/NV21 is listed first, YV12/NV21 will be the default format, and opening video preview will fail due to unsupported format error in the attempt of creating converter (as the renderer does not support YV12/NV21 either).

Previously, tickets #657, #1644, and #1764 allowed invite session to follow SDP answer changes in forking scenario (i.e: when responses have different To tags). This ticket expands the behavior to non-forking scenario, as long as the previous 18x response is not reliable (i.e: using 100rel), this should be inline with ​RFC 6337 section 3.1.1.

This new behavior can be turned off via compile-time setting PJSIP_ACCEPT_MULTIPLE_SDP_ANSWERS or run-time setting pjsip_cfg()->endpt.accept_multiple_sdp_answers. Application can inspect the new flag inv->updated_sdp_answer to check if the SDP negotiation was done with an initial or an updated SDP answer. Furthermore, the existing flag inv->following_fork can be used for checking whether the SDP answer update was on forking scenario.

Thanks George Joseph for the feedback and the patch.

Program received signal SIGSEGV, Segmentation fault.

0x0000000000429047 in pjsua_call_hangup (call_id=2, code=0, reason=0x0,
    msg_data=0x0) at ../src/pjsua-lib/pjsua_call.c:2370
2370   if (call->inv->role == PJSIP_ROLE_UAS)

Step to reproduce:
Run pjsua with dummy TURN server.
--use-ice --use-turn --turn-srv 8.8.8.8:12345 --turn-user na --turn-passwd na
then make call and immediately hangup

If ICE setup takes a long time, for example when using a non-responsive TURN server, call->inv hasn't been created yet, thus causing the crash.

Analysis: Before the crashing line, there's a conditional statement:

    if ((call->med_ch_cb && !call->inv) ||
	((call->inv != NULL) && (call->inv->state == PJSIP_INV_STATE_NULL)))

So, it's possible to enter the block with call->inv == NULL, however later we immediately access call->inv->role, thus causing the crash.

Thanks to Håkan Berg for the report.

This ticket is an initial implementation of ​RFC 4585:

1. PJMEDIA
   • Interworking and coexistence of AVP & AVPF: modify SDP negotiation, transport proto checks in media transports, stream info, etc
   • Implement RTCP-FB packets (generic NACK, NACK-PLI, SLI, RPSI) generation APIs.
   • RTCP-FB support in SDP (​https://tools.ietf.org/html/rfc4585#section-4).
   • Parse RTCP-FB info from SDP and add the info into pjmedia_stream_info for RTCP-FB operational in the stream.
   • Implement RTCP-FB functionality in audio stream, only generic NACK will be implemented.

2. PJSUA/PJSUA2
   • Add RTCP-FB setting structure into account config.

Monitoring RTCP-FB event in application

Application can monitor RTCP Feedback events by implementing PJSUA callback on_call_media_event or PJSUA2 callback Call::onCallMediaEvent(). Sample code for PJSUA:

static void on_call_media_event(pjsua_call_id call_id,
                                unsigned med_idx,
                                pjmedia_event *event)
{
  if (event->type == PJMEDIA_EVENT_RX_RTCP_FB) {
    /* Incoming RTCP-FB event */
    pjmedia_event_rx_rtcp_fb_data *fb_data = (pjmedia_event_rx_rtcp_fb_data*)
                                             event.data.ptr;
    if (fb_data->cap.type == PJMEDIA_RTCP_FB_NACK && fb_data->cap.param.slen == 0)
    {
      /* Generic NACK */
      /* NACK message can be accessed via 'fb_data->msg.nack' */
      ...
    }
  }
}

Currently the resolver releases the mutex when invoking callback, which may cause crash in these scenarios:

• destroy from within callback context,
• destroy while callback is being executed (from any context).

The crash can be avoided by replacing the mutex with group lock (thanks to reference counter functionality provided by the group lock).

Also note that the resolver docs recommends to destroy-recreate the resolver to overcome growing memory problem (see section "Resolver Limitations" in ​here). With the potential crash scenarios described above, it may not be a simple task for application to destroy the resolver without stopping ioqueue & timer first.

Sometimes, it is desirable to delay answering incoming INVITE/re-INVITE, when: A. application needs to perform certain setup which may take some time B. the preparation affects the SDP answer. An example of this is if the application uses third party media, thus it needs to setup the third party media and put the media info in the SDP.

This ticket will add the following:

• Callback on_call_rx_reinvite() in pjsua, and onCallRxReinvite() in pjsua2, for app to receive notification of incoming re-INVITE and decide if it wants to answer asynchronously at a later timing.
• API pjsua_call_answer_with_sdp() for pjsua. And add sdp parameter in CallOpParam for pjsua2, to be used in Call::answer().

The pj_grp_lock_dump() locks the same lock it is dumping and at the end releases it. Unfortunately, the release also call pj_grp_lock_dec_ref() on the lock which calls pj_grp_lock_dump() and ends up in infinite loop.

Thanks Imad Khazali for the report.

To maintain existing calls in IP change scenario, re-INVITE needs to be sent (e.g: for updating Contact header and reinit media). Unfortunately when a call belongs to a non-registering account, such re-INVITE is never sent.

Decoding fmtp is not removed even though Opus config has been changed.

After app calls pjmedia_codec_get_default_param() which will generate default decode fmtp as well, changing the config by calling pjmedia_codec_opus_set_default_param() currently can only add/change the fmtp, but not remove the ones that are not necessary. For example, enabling CBR, then disabling it, will still have the fmtp "cbr=1".

Currently local account is always bound to a transport since r5784 (of 2.8). Unfortunately the current account matching algo in pjsua_acc_find_for_incoming() may select a local account with incompatible transport type, which may cause failure in sending response with error PJSIP_ETPNOTSUITABLE.

If account is enabling NAT64 in its config, trying IPv4 STUN resolution is not necessary and may cause unwanted delay.

Note: the fix in this ticket doesn't affect STUN resolution that is not initiated by an account or a call.

pjsua_core::resolve_stun_server                                             .... Ignoring STUN resolution failure (by setting)
  pjsua_core::pjsua_resolve_stun_servers
    pjsua_core::resolve_stun_entry                                          .... Trying STUN server IPv4 (1 of 1)
      stun_sock::pj_stun_sock_create
      stun_sock::pj_stun_sock_start
        srv_resolver::pj_dns_srv_resolve                                     .... Starting async DNS A query_job 
              resolver::pj_dns_resolver_start_query                              .... Picked up DNS A record from cache, ttl=616663478
                stun_sock::dns_srv_resolver_cb /* called sync. since DNS record is cached*/
              stun_sock::get_mapped_addr
                    stun_session::pj_stun_session_create_req
                    stun_session::pj_stun_session_send_msg                        .... Error sending STUN request: Network is unreachable
                      stun_transaction::pj_stun_client_tsx_create                 .... STUN client transaction created
                          stun_transaction::pj_stun_client_tsx_send_msg               .... STUN sending message (transmit count=1)
                            stun_transaction::tsx_transmit_msg                        .... STUN error sending message: Network is unreachable
                          stun_session::pj_stun_msg_destroy_tdata
                            stun_session::destroy_tdata                               .... tdata 0x1030f24a8 destroy request, force=0, tsx=0x1030f2630
                              stun_transaction::pj_stun_client_tsx_schedule_destroy   .... STUN transaction  0x1030f2630 schedule destroy
                    stun_sock::sess_fail                                          .... Session failed because STUN Binding request failed: Network is unreachable
                      pjsua_core::test_stun_on_status                             .... STUN resolution failed: Network is unreachable

/* pjsua_resolve_stun_servers Loops for 64 sec before Timeout */
 stun_sock::pj_stun_sock_destroy                                                  .... STUN sock 0x103022e28 request, ref_cnt=3
   stun_session::pj_stun_session_destroy                                   .... STUN session 0x102918228 destroy request, ref_cnt=3

The issue is caused when DNS resolution (pj_dns_srv_resolve() ) in pj_stun_sock_start() returns synchronously and the DNS resolution callback has been called. However, there is currently no mechanism to propagate the error from inside the callback to stun_sock, thus pj_stun_sock_start() will always return PJ_SUCCESS. This causes pjsua STUN server resolution to continue waiting until it times out.

(Note that this issue first appears because of ticket #1962, which directly returns without setting any error status after synchronous failure, to prevent double destruction.)

Thanks to Imad Khazali for the report and analysis.

• Recent changes may have increased SIP message size (perhaps mainly from increased SDP size) and when the SIP message size exceeds ~1300 bytes, the SIP message will be sent via TCP transport (if TCP transport is available). Unfortunately, some test components, e.g: SIPp, are configured to use UDP only, so some Python script tests will fail.

From ​https://tools.ietf.org/html/rfc5245#section-15.4

This means that the ice-ufrag
  attribute will be at least 4 characters long, and the ice-pwd at
  least 22 characters long

This ticket will also separate the compile time settings for ice-ufrag length (default still 8) and ice-pwd length (default is 24).

INVITE session (pjsip_inv_session) pjsip_tx_data field (last_answer) is an object shared with pjsip_transaction.last_tsx.

Since it's a shared object, modifying it might trigger a deadlock. The deadlock can be prevented by cloning the pjsip_tx_data and avoid the use of shared object.

Here is the link to the original report:

• ​https://issues.asterisk.org/jira/browse/ASTERISK-27966

Thanks to Richard Mudgett for the patch.

Reported error logs:

Failed creating STUN transport #1 for comp 1: gethostbyname() has returned error (PJ_ERESOLVE)

Failed creating STUN transport #1 for comp 1: Not found (PJ_ENOTFOUND)

The PJ_ERESOLVE error seems to be caused by IPv4 address being resolved to IPv6 address, note that when configured STUN server has IPv4 and it works, only the IPv4 STUN server address will be used.

While the PJ_ENOTFOUND error seems to be caused by failure in host interface enumeration (pj_enum_ip_interface()).

So, in generating STUN & host candidates, I think we should be more forgiving on errors (e.g: skipping STUN Binding resolution when STUN server resolution fails, use default address when host interface enumeration fails).

Thanks Oded Arbel for the report.

Currently, video port's renderer frame buffer is allocated according to the size of the video device's (i.e. the renderer's) parameters, while actually it should be based on the frame size given by video stream instead.

SYSTEM_CLOCK will stop when the device is in deep sleep (on mobile, this can be achieved by pressing the screen lock key and having no app running in background) (see ​here).

This is similar to ticket #1961 for Android.

This ticket will add a timeout (compile time setting PJSIP_TCP_INITIAL_TIMEOUT) to disconnect a TCP server connection if it doesn't receive any data following a successful connect.

Note that when a TCP server connection is idle or not referred anymore, idle timeout will disconnect the connection, refer to PJSIP_TRANSPORT_SERVER_IDLE_TIME

Thanks to Peter Koletzki for the suggestion.

It seems to be needed by AudDevManager::setEcOptions().

Thanks Jure Erznožnik for the suggestion.

Thank you Jure Erznožnik for the suggestion.

AEC info will be added in pjsua_call_dump(). There's also a new API pjsua_get_ec_stat() to query the AEC statistics.

Currently PJSUA media (i.e: internal state call->media) is not yet updated when on_stream_created() is called, as the media update processing is done on provisional media (i.e: internal state call->media_prov) instead. This will cause PJ_EINVAL returned when invoking any PJSUA API that uses call->media, such as pjsua_call_get_stream_info(), from within the on_stream_created() callback.

Thank you Dmytrii Gonchar for the report.

When sending re-INVITE with PJSUA_CALL_REINIT_MEDIA set, the media will reinitialize and the call media order might be different with the media SDP order.

Currently, on re-INVITE with PJSUA_CALL_REINIT_MEDIA the call media is rearrange and later the media order on SDP might get change on pjmedia_sdp_neg_modify_local_offer2().

This will lead to PJMEDIA_EINVALIMEDIATYPE when calling pjmedia_stream_info_from_sdp().

This ticket will avoid rearrangeing media when reinitializing media.

Miscellaneous updates and fixes

OAuth 2.0 is described in ​RFC 6749.

This ticket will add parsing support for the OAuth 2.0 authorisation in PJSIP level.

Currently, TCP/TLS connection will be reused if it's used to send messages to the same destination address. However, there are cases where this may not be desirable, such as for Google Voice account which requires different accounts to use different connections.

This ticket will add option to disable transport connection reuse. But this setting is currently only available in PJSIP level (has not been exported to PJSUA yet).

This ticket will add new callback (on_accept_fail_cb()) to notify when accept operation fails on TLS listener.

Thanks to Peter Koletzki for the suggestion

This ticket will call tp_drop_data_cb() when there is PJSIP_EMISSINGHDR or PJSIP_EINVALIDSTATUS error on incoming message.

Thanks to Peter Koletzki for the report.

Currently video stream will assume a video frame is available and ready to decode when it sees RTP timestamp changing in the jitter buffer (note that RTP packets belong to same a video frame have same RTP timestamp). And when it sees a 'ready' frame, video stream will immediately pull those RTP packets from the JB and start decoding them.

Unfortunately such frame availability check method may cause video quality degradation when the network suffers from consistent RTP reordering, e.g: an RTP packet of a newer frame received before the last RTP packets of an older frame, so the video frame will be decoded without one or more RTP packets.

This ticket updates the behavior by introducing some delay in video frame decoding. The minimum delay is configurable via PJMEDIA_VID_STREAM_DECODE_MIN_DELAY_MSEC, which by default is 100ms. So video stream will make sure that there should be already few video frames in the JB before start decoding the oldest frame.

Thanks Sai Krishna Reddy Kovvuri for the report.

Related to ticket #1793: Avoid unwanted call unhold when sending re-INVITE/UPDATE with call setting param set to NULL

The same behavior is observed when sending re-INVITE upon IP change scenario.

Reported that there are object slicing issues in AudioDevInfo/VideoDevInfo when storing video/audio format detail info (i.e: MediaFormatVideo/MediaFormatAudio) into MediaFormatVector, this is due to the definition of MediaFormatVector that is a vector of MediaFormat (no pointer):

typedef std::vector<MediaFormat> MediaFormatVector;

This ticket will replace MediaFormatVector with MediaFormatVideoVector and MediaFormatAudioVector.

Thanks Ryan Wallach for the report and the suggestion.

Ticket #865 introduces a new callback rtp_cb2(), so we need to consider this when implementing our media transport adapter.

Current PJSUA2 call transfer scenario, when incoming transfer request (i.e: REFER) is received, the new outgoing call will share the same Call instance (with its call ID is switched back and forth between callbacks) until the transfer process is completed.

This ticket will allow application to instantiate a new Call for the new outgoing call in transfer scenario:

class MyCall : public Call
{
private:
  MyAccount *myAcc; // Account derived class

...
public:
  void onCallTransferRequest(OnCallTransferRequestParam &prm)
  {
    ...
    /* Create new Call for call transfer */
    prm.newCall = new MyCall(*myAcc);
  }
};

Thanks Ryan Wallach for the feedback.

Currently libRegisterThread() is not thread-safe. The access to the thread descriptor map needs to be protected in order to avoid data race.

The issue is caused by two things:

• Due to #2097, pjmedia_transport_media_stop() for UDP transport will call pj_ioqueue_post_completion(). This will trigger on_read_complete() callback, i.e. on_rx_rtp() which will call pj_ioqueue_recvfrom(). So the read operations never really stop.
• Ioqueue itself never checks when an op_key is already in the list. Thus, when the media transport is restarted, it will call pj_ioqueue_recvfrom() again. Calling recv() with the same op_key will cause the list to be in a bad state (i.e. the element's next and prev pointers will lose track of its neighbours, hence causing the list to be uniterable). Thus, when later calling pjmedia_transport_detach(), and UDP media transport calling pj_ioqueue_post_completion() for its pending write_op, the function may get stuck when iterating the read list.

It is reported that a deadlock might occur when handling IP change / pjsua_handle_ip_change() is called.

Scenario:

1. [MainThread] Transport is being shutdown e.g: by pjsua_handle_ip_change() -> pjsip_transport_shutdown()
   • at the start it will lock the transport (tp->lock)
2. [pjsua_0] Incoming new INVITE message
   • lock pjsua on pjsua_call_on_incoming()
   • try to lock transport on tsx_update_transport() -> pjsip_transport_add_state_listener()
3. [MainThread]
   • try to lock pjsua_acc_on_tp_state_changed() >> deadlock

When pjsua_handle_ip_change() is called, update contact might be done after listener restart. If listener restart fail, then updating contact should not be attempted since it will also fail.

Reported that when calling a video echo server and video is switched off and on repeatedly every ~10s, sometimes there are issues like:

• taking a while for incoming video echo to be shown after video restart,
• frame rate hiccups after video restart, e.g: a video frame is shown, but then stopped still for sometime, then resumed with higher frame rate (catching up the delay), and eventually played at normal frame rate.

Log file showed strange frame rate change events:

Decoding format changed: 352x288 I420<- 90000/431010(~0)fps
Decoding format changed: 352x288 I420<- 90000/365760(~0)fps
...or..
Decoding format changed: 352x288 I420<- 90000/2610(~34)fps
Decoding format changed: 352x288 I420<- 25/1(~25)fps

Currently the video stream continuously performs frame rate detection, and it should only publish a format-changed event when frame rate is increasing. But the log shows that it also publishes a format-changed event even when it sees frame rate down to zero, which should not happen.

Thank you Giorgio Alfarano for the report.

Based on the ​RFC 3265:

A NOTIFY request is considered failed if the response times out, or a non-200 class response code is received which has no "Retry-After" header and no implied further action which can be taken to retry the request (e.g., "401 Authorization Required".)

However, our current implementation terminates the subscription for ALL non-200 with no retry-after, even for 401/407, which shouldn't be the case.

Currently, there are a couple of problems when making video only call (aud_cnt = 0) with pjsua:

• sound device still opened
• incoming video-only call will be rejected with 488

This ticket will fix these issues.

Reported that WMME audio device thread could be suddenly stopped in a long duration call (about 8 hours or so). Unfortunately it is quite difficult to investigate the cause. So a workaround is to notify app when it happens so app can restart the audio device.

Thanks Mario Ban for the report.

As stated in ​rfc3261

13.2.2.1 1xx Responses

   Zero, one or multiple provisional responses may arrive before one or
   more final responses are received.  Provisional responses for an
   INVITE request can create "early dialogs".  If a provisional response
   has a tag in the To field, and if the dialog ID of the response does
   not match an existing dialog, one is constructed using the procedures
   defined in Section 12.1.2.

   The early dialog will only be needed if the UAC needs to send a
   request to its peer within the dialog before the initial INVITE
   transaction completes.  Header fields present in a provisional
   response are applicable as long as the dialog is in the early state
   (for example, an Allow header field in a provisional response
   contains the methods that can be used in the dialog while this is in
   the early state).

And ​rfc3311:

   When a UAS compliant to this specification receives an INVITE request
   for a new dialog, and generates a reliable provisional response
   containing SDP, that response SHOULD contain an Allow header field
   that lists the UPDATE method.  This informs the caller that the
   callee is capable of receiving an UPDATE request at any time.  An
   unreliable provisional response MAY contain an Allow header field
   listing the UPDATE method, and a 2xx response SHOULD contain an Allow
   header field listing the UPDATE method.

This patch will update invite options when Allow UPDATE header is included in the provisional response of the INVITE.

Thanks to Pirmin (pirmin@walthert.net) for the patch

Changeset r5828 introduced a bug, which allow the library to send 488 response to a Re-INVITE with no SDP when using pjsua2 API.

Currently on_rx_reinvite() will always be set to pjsua_call_on_rx_reinvite() when using pjsua2 API. The callback didn't provide the SDP answer which is expected. If the answer is not set. the library will detect this as an invalid operation and generate the 488 response.

This ticket will avoid the check when no SDP is provided and allow the library to generate the offer instead.

Thanks to Ryan Wallach for the report.

The OpenH264 decoder may return non-dsErrorFree and still produce a frame usually with some noise. This ticket will filter the frame quality by inspecting the return code of the decoder, i.e: drop the frame if some specific error codes are returned (dsRefLost dsNoParamSets, dsDepLayerLost).

The cancel() inside cancel_timer() returns zero when a timer entry is invalid or its expiration callback is being invoked (as the entry is removed already). Unfortunately the cancel_timer() does not check the count returned by cancel() and always proceed further with pj_grp_lock_dec_ref().

This is a bad news if at the same time the entry callback is being invoked by pj_timer_heap_poll() as pj_grp_lock_dec_ref() will be called after the callback returns. Note that the pj_grp_lock_dec_ref() may cause group lock ref count reaching zero, where all member objects of the group lock will invoke their destructors (usually freeing up any memory allocations), so the next pj_grp_lock_dec_ref() will access invalid/freed memory address.

Thanks Keerthi Kumar Thovi for the report and the analysis.

Ticket #1916 enables on_incoming_call() called from on_create_media_transport() for PJSUA2.

However, after the on_incoming_call() invocation, there is a chance that the call is disconnected by the library (e.g: due to unsupported codec or unsupported SIP capability requirement). Unfortunately, there is a bug in the library that application will not be informed about the call disconnection in such scenario.

The patch will add check for codec and capability support from the SDP before calling on_incoming_call(). In this case, when the check fails then on_incoming_call() will not be called.

However, actually there is still a chance of failure in further step of call verification, so the call may get rejected by the library after incoming call callback is invoked. In this case, call state callback on_call_state() will be invoked to notify application about call disconnection.

Thanks to Ryan Wallach for the report.

When using GnuTLS, tls_init() will save cipher suite info to local array and might raise out of bound error if GnuTLS has more supported cipher suite then the maximum array size (MAX_CIPHERS=100).

This patch will not allow adding more cipher once the local array is full.

Thanks to Hugo Lefeuvre for the report and initial patch.

Scenario:

1. send INVITE
2. recv 200/OK
3. app implements on_send_ack() and calls pjsip_inv_create_ack(), but have not sent it.
4. recv retransmission of 200/OK
5. PJSIP auto answers the ACK

The doc of on_send_ack() says:

     * Application creates the ACK request
     *
     * Once it has sent the ACK request, the framework will keep 
     * this ACK request in the cache. Subsequent receipt of 2xx response
     * will not cause this callback to be called, and instead automatic
     * retransmission of this ACK request from the cache will be done
     * by the framework.

But actually pjsip_inv_create_ack() is the one that saves inv->last_ack. So the auto transmission already happens in inv_send_ack() of sip_inv.c, even though app hasn't sent the ACK.

There have been some reports about crash in timer heap (there was #2172, but reported that crash still occurs even after integrating the patch). Creating a stress test for timer heap to reproduce the issue could be useful in identifying the problem and verifying a future solution.

Few notes after running the stress test

• The stress test could reproduce the previously suspected issue of "pop_freelist()+push_freelist() after remove_node() in pj_timer_heap_poll()", here is a bit more description on it:

pop_freelist() will only preserve the timer_id, not the timer entry as the timer entry node has been completely detached from the heap. it may cause crash, as the number of free timer_id is less than timer heap slot, e.g: if the app has 4 timer polling threads, there can be 4 less free timer_id than timer heap slot, while the timer heap slot will grow only when there are 2 free timer heap slot left.

So r5934 removes pop_freelist() + push_freelist() after remove_node().

• The issue described in #2172 (double destroy after cancel_timer() return code not checked) could not be reproduced, but later after a further investigation, found out that it may actually be a false alarm because the poll() sets timer entry group lock to NULL (within the same timer lock block as remove_node()) before invoking callback, so it should be safe from the double destroy issue.

Known issues with current configure-android:

• Detected TARGET_HOST 'llvm' is not recognized by config.sub.
• C++ STL backend uses gnustl_static (llvm does not support it?).
• The default TARGET_ABI, i.e: 'armeabi', is removed since NDK r17.

Thank you Ryan Wallach for the feedback.

Call stack trace:

1  pj_strstr (string.c:154)
2  get_cn_from_gen_name (ssl_sock_ossl.c:1390)
3  get_cert_info (ssl_sock_ossl.c:1455)
4  update_certs_info (ssl_sock_ossl.c:1564)
5  on_handshake_complete (ssl_sock_ossl.c:1590)

pj_strstr() does not expect NULL input string, which is what happens when subject/issuer line is empty.

Thanks Guy Mininberg for the report and the analysis.

Zeroing our buffers should be sufficient as we cannot really manage the OpenSSL internal buffers. Moreover, it seems that OpenSSL already does wipe out its internal buffers, i.e: a lot of OpenSSL_cleanse() calls in OpenSSL source code, the function will fill a buffer with garbage or zero. But unfortunately cannot really find official docs about it.

Additionally, SSL socket pool content will be zeroed before released, it is done using a new API pj_pool_secure_release().

Thanks Peter Koletzki for the feedback.

There are currently a lot of duplication in the SSL backend implementation, which causes major issues, maintenance difficulties, as well as unnecessary complexity when trying to add a new SSL backend.

The major issues are primarily due to revision differences (one backend (OpenSSL) gets updated/fixed a lot, while the other (GnuTLS) lags way behind). These create behavioral differences, where new features such as the new callback on_accept2() is only available for OpenSSL, and potential security problem, since bug fixes are only applied to one backend, while leaving the others exposed.

Thus refactoring is necessary, to make sure that shared codes are put in a separate file.

This ticket will implement video conference using centralized approach which is very similar to the existing audio conference, i.e: there will be an endpoint that act as the conference manager that is capable to establish multiple video calls, mix video from the participants (managed via connecting/disconnecting ports), and then send the mixing result to each participant. The video mixing operation is done by video conference bridge, which will use very similar APIs to the audio conference bridge.

Video conference bridge specifications

• Video timing clock, just like audio, the video conference bridge will have a clock that schedules video flow between ports.
• All registered ports must be in passive mode (at least for now), so the conference bridge will actively pull video frames from source ports and push video frames to sink ports.
• Ports can be bidirectional (they are PJMEDIA ports anyway) and just like in audio, ports connection is unidirectional, from a source port to a sink port.
• A source port may send to multiple sink ports (splitting), and a sink port may receive from multiple source ports (mixing). For splitting, there is no sink port number limitation, but for mixing currently it will only support up to four video sources (actually no limitation, but only the first four will be rendered).
• Video mixing layout. For now it will be static and no customization, e.g:
  • two source ports will be rendered into left/right layout for a landscape sink port, or top/bottom layout for a portrait sink
  • three source ports will be rendered into left+(top/bottom) layout for a landscape sink, or top+(left/right) layout for a portrait sink
  • four source ports will be rendered into 2x2 layout (equally divided slices) for a landscape sink, or four vertically stacked slices for a portrait sink.

Implementation

Basically, the video conference implementation shares the same concept as the audio conference, so for applications that already have audio conference feature, integrating video conference feature should be relatively simple.

For participant, no modification is needed, it will see the video conference as a usual P2P video call, the video from other participants will be rendered into the same video window of the remote side of the video call (the conference manager).

For conference manager, it can setups the video conference using a very similar way as setting up an audio conference (with additional efforts for managing the UI perhaps). A sample code for this can be checked below.

Conference ports in a video window in PJSUA

A video window may be related to two or more conference ports, application should be aware of this so it can manage the conference properly.

A video window represents a video renderer device and is automatically registered to the video conference bridge. The conference slot ID can be queried via pjsua_vid_win_get_info(), the slot ID should be in pjsua_vid_win_info.slot_id. This is a sink port.

As a sink port, it normally has a source, for example a capturer device or a call video stream. The conference slot ID of the source port should be queried separately, for example:

• For capture device, use pjsua_vid_preview_get_vid_conf_port(), the corresponding video window can be queried using pjsua_vid_preview_get_win().
• For call video stream (first video stream), use pjsua_call_get_vid_conf_port(), the corresponding video window can be queried using pjsua_call_get_vid_win(). Alternatively, those info can also be queried via call info, i.e: pjsua_call_info.media[i].stream.vid.enc_slot/dec_slot, the corresponding video window is pjsua_call_info.media[i].stream.vid.win_in.

Sample usage: three parties video conference

Here are steps to setup a three parties video conference:

1. Make two video calls as normal to two other participants.
2. After all calls are established, simply connect those port IDs using pjsua_vid_conf_connect() for PJSUA or VideoMedia::startTransmit() for PJSUA2, e.g:
   • using PJSUA:

     /* Get video ports of call 1, there are two ports as in a video call as
      * encoding and decoding directions may not use the same frame rate or 
      * resolution.
      */
     call_1_dec_port = pjsua_call_get_vid_conf_port(call_1_id, PJMEDIA_DIR_DECODING);
     call_1_enc_port = pjsua_call_get_vid_conf_port(call_1_id, PJMEDIA_DIR_ENCODING);
     
     /* Get video ports of call 2 */
     call_2_dec_port = pjsua_call_get_vid_conf_port(call_2_id, PJMEDIA_DIR_DECODING);
     call_2_enc_port = pjsua_call_get_vid_conf_port(call_2_id, PJMEDIA_DIR_ENCODING);
     
     /* Connect video ports of call 1 and call 2.
      * Note that the source is the stream port in decoding direction,
      * and the sink is the stream port in encoding direction.
      */
     status = pjsua_vid_conf_connect(call_1_dec_port, call_2_enc_port, NULL);
     status = pjsua_vid_conf_connect(call_2_dec_port, call_1_enc_port, NULL);
     

   • using PJSUA2 (exception handling excluded):

     /* Get video ports of call 1 */
     VideoMedia call_1_dec_port = call1->getDecodingVideoMedia(-1);
     VideoMedia call_1_enc_port = call1->getEncodingVideoMedia(-1);
     
     /* Get video ports of call 2 */
     VideoMedia call_2_dec_port = call2->getDecodingVideoMedia(-1);
     VideoMedia call_2_enc_port = call2->getEncodingVideoMedia(-1);
     
     /* Connect video ports of call 1 and call 2 */
     VideoMediaTransmitParam transmit_param;
     call_1_dec_port.startTransmit(call_2_enc_port, transmit_param);
     call_2_dec_port.startTransmit(call_1_enc_port, transmit_param);
     

3. At this point, each participant should be able to see video from the two other participants. The caller will see them in separate windows as it has two video calls (may be combined if preferred, see below), while the other two will only have one incoming video window as usual, just it contains a mixed video (from the caller and the other participant).
4. On the caller side, it may combine video window of call 1 and call 2 into a single window (for simpler UI management perhaps), e.g:
   • using PJSUA:

     pjsua_vid_win_id wid1, wid2;
     pjsua_vid_win_info win2_info;
     
     /* Put incoming video stream from call 1 into call 2 window */
     wid2 = pjsua_call_get_vid_win(call_2_id);
     pjsua_vid_win_get_info(wid2, &win2_info);
     pjsua_vid_conf_connect(call_1_dec_port, wid2_info.slot_id, NULL);
     
     /* Now hide the video window of call 1 */
     wid1 = pjsua_call_get_vid_win(call_1_id);
     pjsua_vid_win_set_show(wid1, PJ_FALSE);
     

   • using PJSUA2:

     /* Function for querying any first video window of the specified call */
     VideoWindow getCallVideoWindow(const Call *call) {
         CallInfo ci = call->getInfo();
         CallMediaInfoVector::iterator it;
         for (it = ci.media.begin(); it != ci.media.end(); ++it) {
     	if (it->type == PJMEDIA_TYPE_VIDEO &&
     	    it->videoIncomingWindowId != PJSUA_INVALID_ID)
     	{
     	    return it->videoWindow;
     	}
         }
         return VideoWindow(PJSUA_INVALID_ID);
     }
     
     /* Put incoming video stream from call 1 into call 2 window */
     VideoWindow wid2 = getCallVideoWindow(call2);
     VideoMediaTransmitParam transmit_param;
     call_1_dec_port.startTransmit(wid2.getVideoMedia(), transmit_param);
     
     /* Now hide the video window of call 1 */
     VideoWindow wid1 = getCallVideoWindow(call1);
     wid1.Show(false);
     

1. In pjsua_acc_del(), it calls pjsua_acc_set_registration(acc_id, PJ_FALSE), which calls pjsua_pres_unpublish(&pjsua_var.acc[acc_id], 0);
2. And then, it will call pjsua_pres_delete_acc() which also calls pjsua_pres_unpublish(acc, flags);
3. It's possible that publish_cb() in (1) is called at the same time as (2) and the callback is in the process of destroying the publish session, hence resulting in a crash.

Reported that on late media offer scenario (receiving INVITE without SDP), dialog is not destroyed properly after the call is disconnected.

After investigation, we found a bug in PJSUA incoming call handler function (i.e: pjsua_call_on_incoming()) that the dialog session previously incremented will never be decremented when the function does not initialize media channel.

Thanks Martin Novotný for the report.

If any capture or playback device ID is set to PJSUA_SND_NULL_DEV (for null audio device), PJSUA will invoke pjsua_set_null_snd_dev() so both device IDs will be reset to PJSUA_SND_NULL_DEV again. Unfortunately PJSUA2 can only configure one audio device (only playback or capture) at one time, so it will never be able to switch over from null device.

This ticket will add support of Darwin native SSL using the Security framework as the SSL backend for Mac OS and iOS.

Video stream keep-alive was disabled in r4120. This ticket will re-enable the mechanism.

This can be useful in situations where video camera access is exclusive, so when we don't need it, the video camera can still be used by the user or other apps.

This ticket will include:

• Avoid creating the video capture if the setting vid_out_auto_transmit is false
• Decrease the video capture reference count when the outgoing transmission is disabled (via the API pjsua_call_set_vid_strm() with op PJSUA_CALL_VID_STRM_STOP_TRANSMIT). When the reference count reaches zero (i.e. no other calls nor the preview window is using it), then the video capture will be released.

If the encoder channel is paused and only decoding allowed, then RTCP RR reports are not being generated.

Currently PJSUA2 internally maintains some lists of objects, e.g: AudioMedia, Buddy, AudioDevInfo, VideoDevInfo, CodecInfo. Unfortunately some APIs for querying and modifying those lists are not really thread safe. For example Endpoint::mediaEnumPorts() will simply return the internal list, and Endpoint::mediaAdd(), Endpoint::mediaRemove(), Endpoint::mediaExists() may access the internal list without mutex protection. Another example is enumeration function such as Aud/VidDevManager::enumDev(), CodecInfoVector &codecEnum() that returns internal list which always be updated at each call of the function, so if two threads call the same function at the same time, one of the thread will get an invalid list. Thank you Niclas Larsson for the report.

Since those PJSUA2 objects are actually just kind of thin wrapper of PJSUA objects, e.g: PJSUA2 AudioMedia is wrapping PJSUA conference bridge port ID, those object should be easily generated from PJSUA. So instead of maintaining list of objects internally, PJSUA2 can simply generate the requested objects on the fly. For example when enumeration function is called, the function will generate the PJSUA2 objects, generate a new list containing those object, and return the list. This way PJSUA2 can also avoid adding another layer of synchronization as PJSUA already provides the thread safety.

This ticket will deprecate these APIs:

• AudioMedia related: Call::getMedia(), AudioMediaVector, Endpoint::mediaEnumPorts(), Endpoint::mediaAdd(), Endpoint::mediaRemove(), Endpoint::mediaExists(), Endpoint::typecastFromMedia(), AudioMediaPlayer::typecastFromAudioMedia(), AudioMediaRecorder::typecastFromAudioMedia(), Endpoint::registerMediaPort()
• Buddy related: BuddyVector, Account::enumBuddies(), Account::findBuddy()
• Codec enumeration: CodecInfoVector, Endpoint::codecEnum(), Endpoint::videoCodecEnum()
• Audio/video device info: AudioDevInfoVector, VideoDevInfoVector, AudDevManager::enumDev(), VidDevManager::enumDev()

This ticket will introduce new APIs:

• AudioMedia related: Call::getAudioMedia(), AudioMediaVector2, Endpoint::mediaEnumPorts2(), Endpoint::registerMediaPort2()
• Buddy related: BuddyVector2, Account::enumBuddies2(), Account::findBuddy2()
• Codec enumeration: CodecInfoVector2, Endpoint::codecEnum2(), Endpoint::videoCodecEnum2()
• Audio/video device info: AudioDevInfoVector2, VideoDevInfoVector2, AudDevManager::enumDev2(), VidDevManager::enumDev2()

Reported callstack trace:

#0  pj_atomic_get (atomic_var=0x0) at ../src/pj/os_core_unix.c:929
#1  pjsip_transport_destroy (tp=0x2a2df78) at ../src/pjsip/sip_transport.c:1309
#2  transport_idle_callback (timer_heap=0x2824280, entry=0x2a2e070)
    at ../src/pjsip/sip_transport.c:1016
#3  pj_timer_heap_poll (ht=0x2824280, next_delay=0x7fa5bd086cb0)
    at ../src/pj/timer.c:650
#4  pjsip_endpt_handle_events2 (endpt=0x2823f58, max_timeout=0x7fa5bd086d00,
    p_count=0x7fa5bd086d18) at ../src/pjsip/sip_endpoint.c:715
#5  pjsua_handle_events (msec_timeout=10) at ../src/pjsua-lib/pjsua_core.c:2134
#6  worker_thread (arg=0x0) at ../src/pjsua-lib/pjsua_core.c:768

The callstack above shows that it was about to destroy transport 0x2a2df78 (note that pj_atomic_get() is just the beginning stage of a transport destroy), while the log shows that the transport had actually been destroyed:

tcpc0x2a2df78  TCP transport destroyed normally

Thanks Adam Wientzek for the report.

Investigation

Both transport destroy operations seem to be performed via timer entry, so the culprit seems to be double timer entry scheduling, which after investigating further the scenario is possible indeed (i.e: timer heap has check for it already, but outside the timer heap lock!). So the fix should be about double timer entry scheduling prevention.

After investigating further, most SIP transport implementations (UDP, TCP, TLS) actually have a group lock for synchronization with socket, it should be a good thing to also integrate the group lock with timer.

Reported that on IP change scenario, local contact may not be updated while it should. After investigation, if force_contact is not configured and the account is not registering, call option PJSUA_CALL_UPDATE_CONTACT will not update the contact.

Thanks Marcus Froeschl for the report.

Scenario:

1. Client SSL socket will send data immediately to the server after it is connected.
2. On the server side, the data sent will be buffered. At this point, the handshake on the server side is not yet completed. When the handshake is complete, the buffered data will not be read immediately. It will not be read unless the client proceed to send data. If this is not the case, the client will likely get a time out to the send operation.

This patch will enable reading the buffered data, immediately after the handshake is complete.