Enhancements to WAV player API at pjsua2, pjsua, and pjmedia:

• API to report file type and size (in samples and bytes):
  • AudioMediaPlayer::getInfo(),
  • pjsua_player_get_info(),
  • pjmedia_wav_player_get_info()
• API to report playback position:
  • AudioMediaPlayer::getPos(),
  • pjsua_player_get_pos()
• EOF callback in AudioMediaPlayer

Also fixed bug in pjsua pjsua_player_set_pos(): in pjmedia the argument is in bytes rather than samples.

Currently, pjsip will use default routing (from OS) when trying to get network interface. In a multihomed environment, sometimes this is not the desired behavior. This ticket will add an option to resolve hostname when trying to get the network interface to be put in the Via or Contact header.

To enable this option set resolve_hostname_to_get_interface from the pjsip_cfg_t::endpt to PJ_TRUE or define PJSIP_RESOLVE_HOSTNAME_TO_GET_INTERFACE as PJ_TRUE on the config_site.h.

Thanks to Alexander Klyushin for the report.

This ticket creates OpenGL 2 framework to be used by other platforms and provides implementation for iOS with OpenGL ES version 2.

IMPORTANT: Starting from release 2.6, OpenH264 1.0 is no longer supported as we move to OpenH264 version 1.6.0 instead. For updated instructions, please see ticket #1947.

Support for the open source implementation of H.264 codec from ​http://www.openh264.org.

Requirement:

• OpenH264 library version 1.0.0 release (download ​here)

Implementation features:

• Constrained Baseline, default level=30 (0x1e)
• packetization modes: 1, 0
• key frame request and indication
• obey remote's fmtp

Notes:

• currently transmission is hard coded to use only single NAL (packetization mode 0) regardless of remote's fmtp for interop reason. The same is also done in ffmpeg wrapper.
• whole frame for file encoding is implemented but currently doesn't work

Below are the steps to use the codec:

For GNU targets:

1. Build the OpenH264 library and install it to a directory (with "make install"). By default it installs to /usr/local. You need to modify the Makefile to install it to different location (=prefix).
2. Run PJSIP's configure. If you specify alternate location (prefix) to OpenH264 installation, specify the location with --with-openh264 option. E.g.:

   ./configure --with-openh264=/home/user/opt
   

3. Add this to your config_site.h:

   #define PJMEDIA_HAS_OPENH264_CODEC		1
   

For Visual Studio:

1. Build the OpenH264 library with MinGW make & gcc, and install it to a directory (with "make install"). By default it installs to /usr/local. You need to modify the Makefile to install it to different location (=prefix).
2. Create openh264.lib from openh264.def, note that the static lib libopenh264.a outputted by MinGW gcc cannot be used by VS, so we are going to use the dynamic lib version instead:
   1. open 'Visual Studio command prompt', e.g: Win Start Menu -> MS VS 2005 -> VS Tools -> VS 2005 Command Prompt
   2. change dir to OpenH264 source root directory
   3. run "lib /def:openh264.def", this will create openh264.lib, copy this file to the OpenH264 installation directory
   4. rename or copy libopenh264.dll to openh264.dll and put it in the system path
3. Add OpenH264 include and library installation paths to Visual Studio search path settings.
4. Add this to your config_site.h:

   #define PJMEDIA_HAS_OPENH264_CODEC		1
   

For iOS:

1. Follow the guidance for GNU targets above.
2. Include libopenh264.a and iOS-SDK's libstdc++.dylib to application project in XCode.
3. Better remove/rename libopenh264.dylib from the OpenH264 installation directory, as somehow it may cause crash on runtime (failed in loading dynamic lib).

For Android:

1. Build the OpenH264 library according to the instructions provided in README.md at the OpenH264 directory. Modify the Makefile to install it to different location (modify the "prefix") and run "make install".
2. Copy libopenh264.so to your Android application library directory (such as libs/armeabi).

Some application want to be able to access TCP socket handle for setsockopt. There's already an API for UDP transport (pjsip_udp_transport_get_socket), and new API is needed for TCP.

iOS video capture and simple renderer were actually already implemented, however it has not been maintained nor tested for long time due to lack of video support on iOS. This ticket will check and update the implementations.

Some additional features planned on the capture device implementation:

• native preview
• enumerate all available capture devices, e.g: front and back camera
• fast switch between capture devices

The simple renderer is a CGImage+UIView based renderer.

Previously, ticket #1644 only allows this to happen upon receiving 183 response only. This ticket will expand it to all 18x responses.

Usage (optional): Append DH parameters into the private key file (privkey_file), for example ​here for ephemeral DH (DHE). Ephemeral ECDH (ECDHE) works automatically. Without specifying a cipher-suite, # openssl ciphers -v DEFAULT is used. Consider reordering or disabling certain suites. Make sure to set the 'method' parameter to the value 'sslv23' because this disables SSL 2.0 and is the only way to enable TLS 1.2 in pjsip, currently. TLS 1.2 is required to enable AES-GCM cipher-suites.

Drawback: For Java clients, go for a 1024bit parameter file, or disable DHE via 'cipher', or put ECDHE high in priority (of 'cipher').

Speed: With a mobile phone from 2006 (Nokia E61), DHE/3DES and a 2048bit parameter, the speed penalty is about 0,5 seconds per connection setup.

Tone generation API in pjsua2 and the SWIG wrappers

Application may want to apply specific socket option to the transport by calling setsockopt.

For TCP and UDP, the options are defined using sockopt_params in pjsua_transport_config.

For TLS, the options are defined using sockopt_params in pjsip_tls_setting.

Currently with OpenSSL, PJSIP uses TLS version 1.0 for both server and client as default. AES-GCM cipher suites require at least TLS version 1.2. In the API of PJSIP, you are able to change to PJ_SSL_SOCK_PROTO_SSL23 which uses SSLv23_method() in OpenSSL.

Although the method name indicates SSL version 2 and SSL version 3, SSLv23_method() is the default in OpenSSL because only that method enables the current, save versions. In OpenSSL 1.0.1, this is SSLv3, TLSv1, TLSv1.1, and TLSv1.2. SSLv2 is turned off (although the method name might suggest the contrary; ​https://www.openssl.org/docs/ssl/SSL_CTX_new.html) Consequently, SSLv23_method() enables PJ_SSL_SOCK_PROTO_SSL23, which enables TLSv1.2, which enables AES-GCM cipher suites.

Because pjsua does not offer a parameter to change the TLS protocol version, the attached patch changes the default of PJSIP to the default of OpenSSL which keeps up with future changes in OpenSSL.

Support for libyuv library which provides YUV scaling and conversion functionality (​https://chromium.googlesource.com/libyuv/libyuv/).

Requirement

• Libyuv library. For building the library, we recommend using the steps described ​here, except for Android target as explained below.

This implementation will provide video scaling and conversion functionality.

Below are the steps to use the codec:

For GNU targets

1. Build Libyuv library. Make sure to execute the Release build step (ninja -j7 -C out/Release). The library will be built on ([libyuv_dir]/out/Release).
2. Run PJSIP's configure and specify the Libyuv folder using --with-libyuv option. E.g:

   ./configure --with-libyuv=/home/user/src/libyuv
   

For iOS targets

1. Build Libyuv library. Make sure to execute the Release build step (ninja -j7 -C out_ios/Release-iphoneos libyuv_unittest). The library will be built on ([libyuv_dir]/out_ios/Release-iphoneos).
2. Run PJSIP's configure-iphone and specify the Libyuv folder using --with-libyuv option. E.g:

   ./configure-iphone --with-libyuv=/home/user/src/libyuv
   

3. Add libyuv.a to application project in XCode. For armv7 architecture, also add libyuv_neon.a to the application project.

Notes for iOS targets:

• To build for armv7s, just follow the instructions for arm64, which will build libyuv for three architectures: armv7, armv7s, and arm64. armv7s it is still not supported because of the limitation from chromium build tools (documented in libyuv's issue ​here).

For Android targets

1. Get libyuv from ​here (tried to build using the official source, but failed).
2. From the libyuv-android directory, run ndk-build. The library will be built on ([libyuv_dir]/libs/armeabi).
3. Create dir out/Release under [libyuv_dir]/jni and copy the libyuv.so there. E.g:

   cd [libyuv_dir]
   mkdir -p jni/out/Release
   cp libs/armeabi/libyuv.so jni/out/Release
   

4. Run PJSIP's configure and specify the Libyuv folder using --with-libyuv option. E.g:

   ./configure --with-libyuv=[libyuv_dir]/jni
   

For Visual Studio

1. Build the Libyuv library.
2. Add Libyuv include and library location paths to Visual Studio search path settings.
3. Add this to your config_site.h:

   #define PJMEDIA_HAS_LIBYUV		1
   

Voice Processing IO Audio Unit is considered to provide nice features such as echo cancellation & AGC, it is supported since Mac OSX 10.7.

Our internal testing indicates that the EC performance is far more superior than any software EC. However, VPIO seems to work only with specific devices, thus requiring you to use the default/built-in audio I/O devices (i.e. it doesn't allow you to select your own devices). Application wishing to disable the hardware EC and use software EC instead can specify PJMEDIA_ECHO_USE_SW_ECHO in pjsua_media_config.ec_options (or MediaConfig.ecOptions in pjsua2).

*Limitation: only works in mono. *

This ticket adds the following APIs:

• Endpoint::libRegisterThread()
• Endpoint::libIsThreadRegistered()

and deprecates the following APIs (experimental and unused):

• Endpoint::libRegisterWorkerThread()
• pjsua_register_worker_thread()

According to RFC 5245 section 12.5: The flows for continued operation, as described in Section 7 of RFC 3725, require additional behavior of ICE implementations to support. If an agent receives a mid-dialog re-INVITE that contains no offer, it MUST restart ICE for each media stream and go through the process of gathering new candidates.

App can modify this behaviour by setting RESTART_ICE_ON_REINVITE to 0 (default is enabled).

Add flag PJSUA_CALL_NO_SDP_OFFER in pjsua_call_flag.

App has no way of specifying desired format and output window when opening preview.

Solution: In Pjsua: Add format and wnd fields in pjsua_vid_dev_preview_param.

Normally the stack will reject any INVITE/reINVITE message with unknown body. This ticket will enable application to process the body, by specifying the new compile time option PJSIP_INV_ACCEPT_UNKNOWN_BODY to PJ_TRUE.

Initial hints for the implementation can be found ​here for OpenSL and ​here for Android JNI.

Take out JNI OnLoad? from Android JNI audio device backend and move it to pjlib, allowing it to be used by other components (or other platforms)

Currently, when application calls pjsua_call_reinvite/update() with NULL call setting param, the existing call settings (including the flag) will be used, this may surprise the application with unwanted actions, for example, when previously application calls pjsua_call_reinvite()/update() with PJSUA_CALL_UNHOLD flag, the flag will stick for any future calls to pjsua_call_reinvite()/update() with NULL call setting param, so at some point it may unhold the call accidentally.

Some flags (such as PJSUA_CALL_UNHOLD, PJSUA_CALL_UPDATE_CONTACT, and PJSUA_CALL_NO_SDP_OFFER) are considered to be one-time operation or not supposed to be done again without explicitly specified in the flag. So when call setting param is NULL, such one-time operation flags are better be reset and application should explicitly specify the flag in the call setting param whenever it needs to release hold, etc.

Thanks to Itay Bianco for the report.

Currently, auto re-registration (e.g: after transport disconnection) interval is configurable via reg_retry_interval/reg_first_retry_interval in the account config, and to avoid all clients re-registering at the same time, the interval will be randomized by +/- ten seconds. Note that the ten seconds random range is hardcoded. While for some cases, application may want the auto re-registration to be done immediately (no such randomized delay). Or even perhaps thinks that ten seconds random range is too tight/loose. This ticket will make the randomized range configurable.

Thanks Johan Lantz for the feedback.

Certain SSL backend implementation, e.g:OpenSSL has feature to enable/disable protocol used. In case of OpenSSL, using SSLv23_method will enable SSLv2-TLSv1.2. Since SSLv2 and SSLv3 are outdated, it will be useful to disable those. This ticket will enable app to specify the protocol used as a combination of pj_ssl_sock_proto value and eventually force client to use the protocol.

Thanks to Thomas Haremsa for the suggestions.

Offering the ability for application to provide a directory where CA certificate files could be found.

Thanks James Cloos and Mark Michelson for the patch.

Add callbacks for application to preview the audio frame traffic from/to sound device. This could be useful for application to be able to inspect signal level and set mic/speaker volume appropriately to avoid clipping or too low/high volume.

Related to #1049:

In PJSUA-LIB:

Sending BYE if call isn't disconnected after CANCEL is issued: it is possible that the CANCEL request and 200/OK to INVITE had crossed each other in the wire; in this case, the call will get CONNECTED and the CANCEL will be rejected with 481.

In registration callback on_reg_state2(), especially in failure cases, sometimes it's difficult to distinguish whether the callback is called for registration or unregistration request.

This ticket will fix it by setting cbparam.expiration field to 0 in case of unregistration and > 0 otherwise.

When creating a pjsip transaction with new group lock, it is not necessary to acquire the lock. An advantage of doing this is prevent the establishment of lock ordering, which can later be picked up by thread detection analysis tool as violation of lock ordering.

This feature will be auto detected and enabled by configure script.

The implementation is using android.hardware.Camera class.

With sample apps and tests focused on Android only.

On IP Address change case, app might want to close the transport before connecting state, however it is not possible to get the transport instance before registration process is finished.

This ticket is mainly for getting registration transport instance before getting the registration result (via pjsua callback on_reg_state).

Example scenario:

1. Connecting to network A (3G)
2. Check if a transport is saved. Finding None so go on.
3. Register requested (and account created)
4. Transport created in PJSIP
5. Connecting to network B (WIFI)
6. Register request on network B (old account destroyed and new one created)
7. Check if a transport is saved. Finding None so go on.
8. Transport A and transport B connect (may happen in an iphone since the 3G connection is never really lost).
9. Transport A is saved and then transport B is saved in its stead
10. If on transport connect we also check if there is already a transport connected and then release the old one (which we are doing at the time), then if by chance transport B connects first, then network A would actually cause a release of transport B which is the latest one
11. Otherwise, transport A is never released.

on_reg_started callback can be used for getting registration transport instance, but we need to include pjsua_internal.h to get account's pjsip_regc instance (to eventually get the registration transport). With this ticket, it can be done as follow:

static void on_reg_started2(pjsua_acc_id acc_id, pjsua_reg_info *info)
{
    pjsip_regc_info regc_info;
    pjsip_regc_get_info(info->regc, &regc_info);

    // transport instance is in 'regc_info.transport'
    ...
}

This ticket will check and fix/implement enumeration of the supported video format in all video capture device implementations.

Currently video renderer is precreated with a specific size and when incoming video frame has been successfully decoded and the size has been different, the decoder will send event that format has been changed and renderer will resize, which may require recreating.

Perhaps we should delay the renderer creation or make sure changing renderer size is a lightweight operation.

Currently, video port has to handle video resizing when video size requested by application is not supported by video device. A raised issue is how to find the closest size from the supported sizes, e.g: should we consider same/closest aspect ratio in the search?

After a discussion, we come to a conclusion that the priority is the closest width, then closest height (when resized with same aspect ratio), and eventually the result may have different aspect ratio. So perhaps we should provide options whether aspect ratio should be maintained and how (crop or pad with black boxes at up&down/left&right sides).

Video port needs to find the best match between requested and supported format. We use format id matching with the nearest width and aspect ratio to find match between the requested format and the supported format.

For format id matching, the priority is:

1. Find exact match
2. Find format with the same color space
3. Use the first supported format.

For ratio matching: Find the lowest difference of the aspect ratio between the requested and the supported format.

Note: ratio matching can be skipped by specifying

#define PJMEDIA_VID_PORT_MATCH_WIDTH_ARRAY_SIZE 1 

This ticket will add support to bdSound's bdiMad v2.0.0. Thanks to Francesco Zocchi for the patch.

This could be useful for testing purposes.

Current PJSIP resolver lacks these functionality:

• NAPTR support
• full IPv6 support
• unspecified SRV lookups

This ticket will enable application define its own resolver implementation and set it to be used instead of the method built in. Application can do this by by defining the callback in pjsip_ext_resolver and pass it to pjsip_endpt_set_ext_resolver() function.

Thanks to Joshua Colp for the suggestion and initial patch.

Add support if the underlying OpenSSL library supports more than 100 suites.

Currently, the pjproject assumes no OpenSSL library with more than 100 cipher suites exists. OpenSSL 1.0.1 in Ubuntu 14.04 has 98. However, the amount of cipher suites raised a lot. If such a library (with more than 100 ciphers) is used, only the first 100 cipher-suites are allowed to be set manually (for example via --tls-cipher in pjsua). If the cipher suites are not changed manually, this code is not passed and the default cipher-suites are offered by pjsua (even if there are more than 128).

The ticket also allows all available cipher suites by changing the filter from "ALL" to "ALL:COMPLEMENTOFALL". The OpenSSL team disabled ciphers without encryption (eNULL). However, those can be useful while testing. This change does not change existing implementations, because the implementation of the method "set_cipher_list" sets the list (implicitly) to "DEFAULT" when the condition "ssock->param.ciphers_num == 0" is met (= no cipher-suites are set manually). DEFAULT is without aNULL and eNULL ciphers.

Special thanks to Alexander Traud for the patch and the ticket description above.

Since March 2015, a TLS/SSL security vulnerability was revealed, which mandates to disable certain TLS cipher-suites <​https://en.wikipedia.org/wiki/FREAK>. Furthermore since February 2015, RFC 7465 prohibits the cipher-suite RC4. To achieve this with OpenSSL, the cipher-suite list has to be changed from "DEFAULT" to "HIGH". Furthermore, all ciphers which were not included in "DEFAULT" before (like eNULL), should not appear now either. Therefore: HIGH:-COMPLEMENTOFDEFAULT.

Special thanks to Alexander Traud for his excellent suggestion, description, and patch.

When a client connects via SSL/TLS, the server uses a RSA key-pair usually. However, more such algorithms exist like DSA and ECDSA, and if the server socket setups a certificate for either one of those, it would loose compatibility to RSA-only clients. This ticket allows the server socket to be configured with up to one RSA, ECDSA and DSA key each. Also, if any socket clients are not compatible with SHA-2 hashed certificates ­for example Nokia mobile phones, the server socket still can use RSA/SHA-1 for legacy clients and ECDSA/SHA-2 for everyone else.

Thanks Alexander Traud for the feedback and the patch.

Currently we use compile-time constants PJSIP_TCP_KEEP_ALIVE_INTERVAL and PJSIP_TLS_KEEP_ALIVE_INTERVAL. This ticket will make the settings configurable during runtime.

Provide a new callback pjsip_tp_on_rx_dropped_cb() which will be called whenever SIP transport drops data with the following information:

/**
 * Structure of dropped received data.
 */
typedef struct pjsip_tp_dropped_data
{
    /**
     * The transport receiving the data.
     */
    pjsip_transport *tp;

    /**
     * The data.
     */
    void	    *data;

    /**
     * The data length.
     * If the status field below indicates an invalid SIP message
     * (PJSIP_EINVALIDMSG) and application detects a SIP message
     * at position p, it can pass the data back to PJSIP to be processed
     * by setting the len to p. This can be useful for apps which
     * wishes to use the same transport for SIP signalling and non-SIP
     * purposes (such as SIP outbound using STUN message).
     */
    pj_size_t	     len;

    /**
     * The status or reason of drop. For example, a leading newlines (common
     * keep-alive packet) will be dropped with status PJ_EIGNORED, an invalid
     * SIP message will have status PJSIP_EINVALIDMSG, a SIP message overflow
     * will have status PJSIP_ERXOVERFLOW.
     */
    pj_status_t	     status;

} pjsip_tp_dropped_data;

With sample apps (pjsua2) and tests focused on Android only.

On re-INVITE (e.g: lock codec, ICE update, session refresh), SRTP sends all crypto-suites instead of just the currently active. In the Sipit31 tests, this seems to be just fine (the same current active cryptos were used on both sides, no RTP protect/unprotect error), but this is perhaps risky, e.g: remote may mistakenly see it as SRTP session restart, thus remote will reset its SRTP states (so RTP seq calculation became out of sync, etc). So sending the current active crypto should be safer.

In addition to checking encoding name and clock rate, pjmedia_sdp_neg_fmt_match() API will check encoding parameters as well.

From: ​https://tools.ietf.org/html/rfc4566#page-25

a=rtpmap:<payload type> <encoding name>/<clock rate> [/<encoding parameters>]

For audio streams, <encoding parameters> indicates the number of audio channels. This parameter is OPTIONAL and may be omitted if the number of channels is one, provided that no additional parameters are needed.

For video streams, no encoding parameters are currently specified.

Currently there is no mechanism to notify other party about new video orientation, so in this ticket, we implement a feature which allows the video sender to transmit video with the "proper" orientation at all time, i.e: head upside whatever the device orientation is. When a device orientation changes, the application can tell the library and the library will rotate the video, resize it accordingly, and insert blank/black in left+right sides. The receiver/decoder side will then receive the video in the correct orientation, nothing to do on the decoder side.

The feature is available on iOS and Android only.

This ticket will add PJSUA2 Video Codec API and Video Device API

Currently pjsua_start() will automatically start NAT type detection, it may block the application UI/main thread (the thread calling pjsua_start(), usually it is the UI/main thread), when the configured STUN server is down.

So, let's just perform NAT type detection only after STUN server resolution is succeeded. Also, make the library starting NAT type detection only when pjsua_config.nat_type_in_sdp is set to non-zero.

Calling #pjsua_init() will initiate an async process to resolve and contact each of the STUN server entries to find which is usable. This callback is called when the process is complete, and can be used by the application to start creating and registering accounts. This way, the accounts can avoid call setup delay caused by pending STUN resolution.

By forcing to destroy the STUN session, it avoids the delay when trying to destroy PJSIP or cancelling STUN server resolution.

Currently each open operation of iOS audio device will reset AVAudioSession category to AVAudioSessionCategoryPlayAndRecord (and without any option), so there is a risk that any audio session settings set by application to be reseted or overridden. To avoid that, the category setting will be moved to audio device factory initialisation.

Provide a compile time option PJSIP_TCP_TRANSPORT_DONT_CREATE_LISTENER and PJSIP_TLS_TRANSPORT_DONT_CREATE_LISTENER for TCP/TLS transport to skip creating the listener. Some applications may not have real use for a listener since they don't need to accept direct incoming connections, hence they can use this option.

This ticket will add support to set video codec param using PJSUA2 API.

Example of use:

  /* MyApp.ep is the Endpoint instance */
  VidCodecParam codecParam = MyApp.ep.getVideoCodecParam("H264");    		
  MediaFormatVideo newFormat = codecParam.getEncFmt();
  newFormat.setFpsNum(30);
  newFormat.setFpsDenum(1);
  codecParam.setEncFmt(newFormat);
  MyApp.ep.setVideoCodecParam("H264", codecParam);

To support to Visual Studio 2015 (VS 2015), this ticket will introduce these files:

- Solution file : pjproject-vs14.sln
- Project files : *.vcxproj
- Project properties sheet files : *.props
- Project filter files : *.filters

Please note that in VS 2015, VC++ directories settings are available in the user property sheet.

To include them in a global setting, use Microsoft.cpp.<Platform>.users (e.g: Microsoft.Cpp.Win32.user) properties sheet file.

How to use WebRTC AEC

To use WebRTC AEC, specify the value PJMEDIA_ECHO_WEBRTC in pjsua_media_config.ec_options (MediaConfig.ecOptions for PJSUA2), and specify the delay estimate for sound card and system buffers in pjsua_media_config.ec_tail_len (MediaConfig.ecTailLen for PJSUA2). WebRTC AEC's performance is very dependent on this delay calculation. A poor estimate, even by as little as 40ms, may affect the echo cancellation results greatly. Our WebRTC wrapper has enabled the delay agnostic feature, which will adjust the delay accordingly, however it may take some time (5-10s or more) for the AEC module to learn the optimal delay, thus a good initial estimate is necessary for good EC quality in the beginning of a call.

The delay should be at least as much as frame time (by default, ptime is 20ms) + PJMEDIA_WSOLA_DELAY_MSEC (the default is 5ms).

In PJSUA sample application, run it with the options --ec-opt=3 and specify the delay estimate in --ec-tail options. For example:

./pjsua --ec-opt=3 --ec-tail=30

Poor WebRTC EC quality

Disable PJMEDIA_WEBRTC_AEC_USE_MOBILE (set it to 0), then change the definition of SHOW_DELAY_METRICS in pjmedia/src/pjmedia/echo_webrtc.c to a non-zero value. For example, if you set to 250, with ptime 20ms, it will print the delay metrics stats every 5 seconds or so.

#define SHOW_DELAY_METRICS	250

What you need to see in the log is the line:

median=%d, std=%d, fraction of poor delays=%f

Fraction of poor delays is a value between 0 and 1. The closer the value to 1, the poorer the EC quality.

To improve the EC quality, here are a few things you can try:

• Adjust the EC tail value to the median value.
• Activate noise suppression (pass the options when creating EC with flag PJMEDIA_ECHO_USE_NOISE_SUPPRESSOR).
• Increase EC aggressiveness (pass the options when creating EC with flag PJMEDIA_ECHO_AGGRESSIVENESS_AGGRESSIVE).
• Disable the mobile version by setting PJMEDIA_WEBRTC_AEC_USE_MOBILE to 0 (by doing this, AEC will operate in floating-point mode (instead of integer), which may decrease performance).

Problem with EC quality on Mac

The problem with echo on Mac and negative delay has been reported by several of our users and seems to be a known issue, as documented in ​Chrome M45 WebRTC Release Notes and ​Issue 3929: AEC on Mac can be put in non-causal state. Unfortunately, there's no known workaround as of now.

*Starting from release 2.6, the build instructions below is no longer required since WebRTC has been integrated into third_party. For more details, please refer to ticket #1954 instead.*

How to build WebRTC

Mac

Refer to WebRTC development guide ​here.

Then configure PJSIP with:

./configure --with-webrtc=/Users/me/webrtc-mac

iOS (ARM)

Refer to WebRTC development guide ​here.

PJSIP's auto configuration will look for the library in out_ios/Release folder, so make sure you set the output dir properly when building !WebRTC

export GYP_GENERATOR_FLAGS="output_dir=out_ios"
ninja -C out_ios/Release

Then configure PJSIP with:

./configure --with-webrtc=/Users/me/webrtc-iphone

Android

Android development is only supported on Linux (reference ​here), so if you use other platforms, you need to set up a Linux virtual machine.

Alternatively, you can build the stripped down version of WebRTC instead, which will only build the required AEC module and its required dependencies. These steps below are tested on a Mac machine:

1. Create a working directory, for example: webrtc-android.
2. Go to the work dir and unzip webrtc-android-jni.zip (provided in the ticket attachment below).
3. Go to jni folder and run ndk-build.
4. Copy the resulting .so files from ../libs/[target_architecture] into your Android application project directory, for example:

   cp ../libs/armeabi/*.so /Users/me/pjproject/pjsip-apps/src/swig/java/android/libs/armeabi
   

5. Add this in PJSIP's user.mak, located inside PJSIP's work dir:

   export CFLAGS += -I"/Users/me/webrtc-android/src" -DPJMEDIA_HAS_WEBRTC_AEC=1 -DPJMEDIA_WEBRTC_AEC_USE_MOBILE=1
   export LDFLAGS += -L"/Users/me/webrtc-android/libs/armeabi" -lwebrtc
   

   Rebuild PJSIP.

For other architectures

(or custom WebRTC installation undetected by configure script)

You can customize PJSIP's user.mak file, located inside PJSIP's work dir:

export CFLAGS += -I"/Users/me/webrtc/src" -DPJMEDIA_HAS_WEBRTC_AEC=1
# Uncomment below if you want to use the lighter, mobile-optimized version
# export CFLAGS += -DPJMEDIA_WEBRTC_AEC_USE_MOBILE=1
export LDFLAGS += -L/Users/me/webrtc/[WebRTC_lib_path] -laudio_processing -lcommon_audio -lsystem_wrappers
# If you build for ARM architecture, uncomment below
# export LDFLAGS += -laudio_processing_neon -lcommon_audio_neon
# Or else, if you build for MIPS architecture, uncomment below
# export LDFLAGS += -laudio_processing_neon -lcommon_audio_neon
# Or else, if your machine supports SSE2, uncomment below
# export LDFLAGS += -laudio_processing_sse2

Then rebuild PJSIP.

Compatibility issue

There is a known compatibility issue with recent WebRTC version, so we'd recommend to use an older WebRTC version, e.g: about October 2015 version.

When receiving re-INVITE without offer, RFC3261 section 14.2 recommends UAS to generate offer with all capabilities that the UA is willing to support to avoid the need for the peer to reject the session description. Some SIP UA (e.g: B2BUA) seem to rely on this recommendation for media switching (e.g: stop IVR and start video).

Currently the behaviour of pjsua is to automatically send offer generated from the current active session, application is not involved in the offer generation. This callback will give the application a better control in the offer generation, for example it may add video via modifying call setting, enable/disable codecs from within the callback, so application could put all its capabilities in the offer.

This ticket will apply the following updates on the ioqueue (most affected implementations are select & epoll):

• Protection against high fds (i.e: fd > FD_SETSIZE) in ioqueue socket registration to avoid potential memory corruption caused by select().
• Added new ioqueue API pj_ioqueue_trylock_key().
• Event dispatcher functions should not wait if an ioqueue key mutex is locked by another thread (perhaps the event for the key is being processed by that thread), skip this key and process next ioqueue key.

Thank you Christian Ambach for the patches.

Some platforms use PortAudio as their default audio device backend, while the PortAudio version we are using is quite old and the effort to update the version is not small. Meanwhile, we already have our own audio device implementations that are considerably lightweight and stable as they have been around and tested for quite long time. So yes, we are considering to update the default audio device backends to our own implementations.

• MacOSX
  Currently the default backend is PortAudio. since MacOSX 10.11, it produces warning message (thank you Martin Ohlsson for the report):

  WARNING:  140: This application, or a library it uses, is using the deprecated
  Carbon Component Manager for hosting Audio Units. Support for this will be
  removed in a future release. Also, this makes the host incompatible 
  with version 3 audio units. Please transition to the API's in AudioComponent.h
  

  So we'll update the default backend to our CoreAudio implementation.

• Linux/*nix
  The default is also PortAudio (ALSA or OSS), we'll use our ALSA implementation (if ALSA header is available).

• MinGW
  The default are both PortAudio and WMME, we'll just use WMME as default.

This ticket will add support to FFmpeg 2.8 with these general modifications:

• Change FFmpeg const using 'AV' previx. e.g: PIX_FMT_NONE -> AV_PIX_FMT_NONE
• Change deprecated method avcodec_get_frame_defaults() -> av_frame_unref()

Application may keep using older FFmpeg version, i.e: the versions that are already supported before the changes in this ticket, by defining macro PJMEDIA_USE_OLD_FFMPEG in the build, e.g: adding this line in config_site.h:

#define PJMEDIA_USE_OLD_FFMPEG

Note: configure script (of GNU autoconf build system) will automatically add this macro when it detects that older FFmpeg version is being used.

Support Windows 10 / Universal Windows Platform and continue the work for Windows Phone 8 support (#1704).

Build guide: Building for UWP & WP8

Opus is a totally open, royalty-free, highly versatile audio codec. Opus is unmatched for interactive speech and music transmission over the Internet, but is also intended for storage and streaming applications. It is standardized by the Internet Engineering Task Force (IETF) as RFC 6716 which incorporated technology from Skype's SILK codec and Xiph.Org's CELT codec. (From ​http://www.opus-codec.org)

How to use:

1. Download Opus source code from ​here (latest stable release 1.1.2 or above is recommended).
2. Build Opus:
   1. For iOS, get the build script ​here or ​here.
   2. For Android, get the Android makefile attached in this ticket (tested with Opus 1.1.2), or check ​here
      • create jni folder in Opus root directory and put Android.mk from this ticket attachment in there
      • run ndk-build from Opus root directory
      • make a new folder called opus-dev-lib and create subfolders: include/opus and lib in it
      • copy Opus header files from [OPUS_ROOT_DIR]/include to opus-dev-lib/include/opus and libopus.a file from [OPUS_ROOT_DIR]/obj/local/armeabi to opus-dev-lib/lib
   3. For GNU build system, e.g: *nix, Linux, MacOS, just invoke ./configure, make, and make install.
   4. For Windows:
      • use GNU build tools such as MinGW, or
      • use VS solution & projects provided by Opus (usually in win32 dir), then modify the following line in the end of pjmedia/src/pjmedia-codec/opus.c appropriately:

         #   pragma comment(lib, "libopus.a")
        

1. Run PJSIP's configure script with the option --with-opus=your_opus_installation_dir. For example on Android platform:

   ./configure-android --with-opus=[absolute-path-to]/opus-dev-lib
   

2. Verify if it's a sucess:

   checking for OPUS installations..
   Using OPUS prefix... [absolute-path-to]/opus-dev-lib/
   checking opus/opus.h usability... yes
   checking opus/opus.h presence... yes
   checking for opus/opus.h... yes
   checking for opus_repacketizer_get_size in -lopus... yes
   OPUS library found, OPUS support enabled
   

Opus Codec Settings

General codec settings for this codec such as VAD and PLC can be manipulated through the setting field in #pjmedia_codec_param (see the documentation of #pjmedia_codec_param for more info).

For Opus codec specific settings, such as sample rate, channel count, bit rate, complexity, and CBR, can be configured in #pjmedia_codec_opus_config. The default setting of sample rate is specified in #PJMEDIA_CODEC_OPUS_DEFAULT_SAMPLE_RATE. The default setting of bitrate is specified in #PJMEDIA_CODEC_OPUS_DEFAULT_BIT_RATE. And the default setting of complexity is specified in #PJMEDIA_CODEC_OPUS_DEFAULT_COMPLEXITY.

After modifying any of these settings, application needs to call #pjmedia_codec_opus_set_default_param(), which will generate the appropriate decoding fmtp attributes.

Here is an example of modifying the codec settings:

   pjmedia_codec_param param;
   pjmedia_codec_opus_config opus_cfg;

   pjmedia_codec_mgr_get_default_param(.., &param);
   pjmedia_codec_opus_get_config(&opus_cfg);
   ...
   // Set VAD
   param.setting.vad = 1;
   // Set PLC
   param.setting.vad = 1;
   // Set sample rate
   opus_cfg.sample_rate = 16000;
   // Set channel count
   opus_cfg.channel_cnt = 2;
   // Set bit rate
   opus_cfg.bit_rate = 20000;
   ...
   pjmedia_codec_opus_set_default_param(&opus_cfg, &param);

Currently pjmedia library will depend on pjmedia-audiodev and pjmedia-videodev, and vice versa. This may cause problem, for example when building shared libraries with linker flags --as-needed.

Currently pjsua will always open both speaker and microphone. Some applications, such as Push-To-Talk, may have microphone off most of the time and prefer to open only the speaker.

While working on pjsua2 sound device, there seem to be a related issue, calling setPlaybackDev() & setCaptureDev() consecutively may not work as expected, i.e: for changing both playback and record devices, perhaps due to asynchronous dev open/close operations, this is experienced on Linux platforms. So, perhaps we should consider to add an API to set both at once, e.g: setPlaybackAndCaptureDev().

Currently, on Android, due to unavailability of native UUID support, we use guid_simple.c which relies on simple rand(). Unfortunately, this may cause ID collision, such as for Call-ID. The proper implementation is to use java.util.UUID API instead.

Thanks to Johan Lantz for the report and the patch.

In the beginning of video call, video keyframe may get lost due to NAT hole punching, probing period in auto-switch to RTP port source feature, or normal packet loss. Mechanism to request video keyframe is currently available, however it takes time. So sending video keyframe regularly at some interval in the beginning of video session may reduce the missing keyframe period. It should be configurable at run-time and compile-time (for default setting).

From our IP Address Change wiki: "If ICE media transport is used, and STUN is enabled on the media transport, then the media transport will automatically update its publicly mapped IP address from the STUN Binding response. The transport should send STUN Binding request periodically (approximately every 15 seconds) as NAT keep-alive mechanism, so the address change will be detected by the transport automatically during this operation."

This ticket is to provide a notification to the app when such address change happens due to the STUN keep alive operation (as indicated by the log saying: STUN mapped address found/changed: [new address here]).

The app will get notified via on_call_media_transport_state() callback and it can inspect ext_info to verify the operation to be equal to PJ_ICE_STRANS_OP_ADDR_CHANGE.

On media stream creation, the stack will call pjstun_get_mapped_addr2() and might fail if the STUN server is unreachable.

Scenario:

1. Start PJSIP with STUN server A, STUN server B (both STUN servers are reachable)
2. PJSIP will use STUN server A
3. STUN server A becomes unreachable, STUN servr B remains reachable
4. User dials a call
5. PJSIP tries to resolve call media with STUN server A, but fails. (PJLIB_UTIL_ESTUNNOTRESPOND)
6. The stack will not fallback to STUN server B nor continue without STUN.

This patch will ignore the error from pjstun_get_mapped_addr2() using pjsua_config::stun_ignore_failure.

Currently there are already pjsip_udp_transport_start() for IPv4 and pjsip_udp_transport_start6() for IPv6, the new API should be a little cleaner as it can create both IPv4 and IPv6 transports.

Also added QoS settings for the transport socket.

Thanks George Joseph for the patch.

Issue:

1. Start PJSIP with STUN server A, STUN server B (both STUN servers are reachable)
2. PJSIP will sue STUN server A
3. STUN server A becomes unreachable, STUN servr B remains reachable
4. User dials a call
5. PJSIP tries to resolve call media with STUN server A, but fails. (PJLIB_UTIL_ESTUNNOTRESPOND)
6. PJSIP does NOT try to fall back to STUN server B (issue)

The ticket will add an API pjsua_update_stun_servers() to update STUN servers list after startup (previously, it can only be specified prior to calling pjsua_init()). Then when call media fails due to pjstun_get_mapped_addr(), we add an option PJSUA_STUN_RETRY_ON_FAILURE in pjsua_acc_config.media_stun_use (by default it's disabled/no retry), which enables retrying other STUN servers when the above scenario happens.

This ticket will contain various patches related to FFMpeg.

For example, newer FFMpeg might deprecate API and introduce a new one.

This ticket will be the placeholder for various fixes/modifications in order to comply with the guidelines/recommendations in Apple's official doc ​Supporting IPv6 DNS64/NAT64 Networks.

This ticket will enable PJSUA application to resolve target hostname to IPv6 address. As currently PJLIB-UTIL DNS resolver has not completely supported IPv6 yet, the resolution will only work when DNS server setting (i.e: pjsua_config.nameserver) is not configured, the library will resolve hostname using pj_getaddrinfo().

Update: Ticket #1927 has been completed, PJLIB-UTIL DNS resolver now supports IPv6 for DNS SRV and DNS A/AAAA resolutions. So the following specifications should be applicable to PJSUA-LIB with and without nameserver settings.

Account transport type

This new field is added to PJSUA account structure (in pjsua internal), it represents account transport preference, i.e: local account will always have specific transport type (as it is created with transport ID param), and normal account will have specific transport type only if it is bound to specific transport, i.e: via pjsua_acc_config.transport_id.

Contact generation

• If account is bound to a specific transport address family of the contact will use the transport address family.
• Otherwise:
  • if target is an IP address, generated contact will use the same IP version as target
  • if target is a hostname (this is rather difficult as we don't really resolve the target here, except for a special case below):
    • if account transport type is specified, generated contact will use IP version as account transport type
    • otherwise, it will simply use IPv4 (!) as there is no target resolution here, hence application is recommended to bind an IPv6-only account to a specific IPv6 transport.
    • special case when pjsua_acc_config.contact_use_src_port is set (by default it is set) and target param transport is TCP/TLS, it will resolve target hostname first, try to connect to that IP using transport with the same IP version as target, and update Contact address with the transport binding address, so if the hostname is resolved to IPv6, Contact address will use IPv6 address.

SIP transport

• If account is bound to a specific transport, it will use that transport
• Otherwise, it will use transport with same IP version as target (resolved address, if it is a hostname)

Media

Media transport is independent from SIP, currently it will simply follow the account setting, i.e: pjsua_acc_config.ipv6_media_use.

Caveat

• When making call using hostname 'localhost' (on Windows 10), it will now resolve to IPv6 address (was to IPv4 address). The solution is to make call using account that is bound to IPv4 transport. The OS seems to by default register 'localhost' as '::1' and '127.0.0.1' and '::1' has higher priority.

This is a continuation of #419.

Backward compatibility break

Field server of pj_dns_srv_record is now using type pj_dns_addr_record which supports both DNS A and DNS AAAA records, previously it was using type pj_dns_a_record which supports DNS A record only.

Starting in Mac OS X 10.9, QTKit is deprecated. To disable QT video capture device, you can set PJMEDIA_VIDEO_DEV_HAS_QT to 0.

This will update pjnath-test for IPv6 support.

• Stun sock test : IPv4 and IPv6 test
• Turn sock test : IPv4 and IPv6 test
• ICE test : IPv4, IPv6, IPv4+IPv6 and mixed test (IPv4 controller v IPv6 controlled)

Some application may process event subscription asynchonously, e.g: save pointer to evsub instance on evsub callback and return immediately from the callback while some other thread is assigned to continue some operation on the evsub instance. Unfortunately, the evsub instance may get destroyed before the other thread starts working on the evsub and later when it starts working on the evsub, the pointer to evsub is already invalid.

This ticket will introduce reference counting APIs so such application could safely keep a pointer to evsub instance.

Thanks George Joseph for the patch.

We recommend libyuv which provides video scaling and conversion method. However, currently we only support libyuv as a stand alone project, with these drawbacks:

• Following the recommended steps to build the library, it is required to also download chromium project which is large in size (approximately 10GB).
• Changes on the libyuv project might break the existing build configuration.

This ticket will add libyuv to the third party folder, use a specific version and include only the necessary source.

Below is the steps to build and use libyuv:

For GNU targets

1. By default, libyuv will always be built automatically. This automatic build can be disabled via configure script param --disable-libyuv or --disable-video.
2. if you want to use external libyuv, specify --with-external-libyuv and set header/library search paths via CFLAGS and LDFLAGS environment variables.

Additional step for iOS targets

1. Add third-party/lib/libyuv-[TARGET].a to application project in XCode.

Known issues

1. GCC 4.2 to 4.7 have a known issue using vectors with const, sample build error message:

   ../../yuv/source/row_common.cc: In function 'void libyuv::YuvPixel(uint8, uint8, uint8, uint8*, uint8*, uint8*, const libyuv::YuvConstants*)':
   ../../yuv/source/row_common.cc:1256: error: invalid types 'const signed char __vector__[int]' for array subscript
   ../../yuv/source/row_common.cc:1257: error: invalid types 'const signed char __vector__[int]' for array subscript
   

   Solution: use GCC 4.8 or newer, or if you don't need video feature, just disable libyuv (i.e: using configure script param --disable-libyuv or --disable-video).

For Visual Studio

1. libyuv will always be built automatically.
2. To enable libyuv, add this to your config_site.h:

   #define PJMEDIA_HAS_LIBYUV		1
   

iOS 10 is coming with new features such as CallKit, framework changes such as in AVFoundation, and also API deprecations. This ticket will review and apply any integrations needed in PJSIP library.

PushKit guide

What your application needs to do:

1. Set up PushKit according to ​Apple's doc.
2. Upon startup, request for push tokens.
3. Send REGISTER to SIP server with the push token info, the info may be added as a specific header, Contact header params, or Contact URI params, to tell the SIP server about the PN (Push Notification) info (server URI, tokens, etc). You can use pjsua_acc_config.reg_hdr_list, pjsua_acc_config.contact_uri_params, or pjsua_acc_config.reg_contact_params (the later is available in ticket #1965).
   Note: ​RFC 8599: Push Notifications in the Session Initiation Protocol (SIP) suggests to put it in Contact URI params, for example:

   Contact: <sip:alice@alicemobile.example.com;
        pn-type=acme;pn-methods="INVITE";
        pn-uri="https://pn.acme.example.com/ZTY4ZDJlMzODE1NmUgKi0K">
   

4. After a successful registration, SIP client can go background and will be woken up by the OS upon receiving push notification.

Note: starting from iOS 13, there's a new requirement: Apps receving VoIP pushes must post an incoming call (via CallKit or IncomingCallNotifications) in the same run loop as pushRegistry:didReceiveIncomingPushWithPayload:forType:[withCompletionHandler:] without delay. Terminating app due to uncaught exception 'NSInternalInconsistencyException', reason: 'Killing app because it never posted an incoming call to the system after receiving a PushKit VoIP push callback.' In order to make it work with the normal SIP flow which may require you to wait for some time to receive the INVITE message, please look at Apple's recommendation in its ​developer forum.

Configure your SIP server:

1. Parse the PN info in registration.
2. Upon receiving an incoming SIP INVITE, SIP server should contact PN server as specified via PN URI and tokens.
3. After some interval (assuming the SIP client has been woken up), it then can send/forward the SIP INVITE message to the SIP client.

CallKit support

​CallKit framework allows apps to use the native phone UI to receive incoming calls and make outgoing calls. In order to achieve this, CallKit requires the call audio to start only when audio session has been activated, thus it's recommended that when using PJSIP, you open the sound device only when necessary. It can be done by:

1. Starting PJSIP with no sound device (by calling pjsua_set_no_snd_dev() after startup).
2. Close the sound device when unused, also with the same API (pjsua_set_no_snd_dev()).
3. Upon audio session activation, open the sound device with the API pjsua_set_snd_dev().

Below is an example on how to integrate CallKit with PJSIP, with the delegate functions taken from ​Speakerbox sample app provided by Apple.

To make outgoing call:

    func provider(_ provider: CXProvider, perform action: CXStartCallAction) {
        /* 1. We must not start call audio here, and can only do so
         *    once the audio session has been activated by the system
         *    after having its priority elevated. So, make sure that the sound
         *    device is closed at this point.
         */

        /* 2. Provide your own implementation to configure
         *    the audio session here.
         */
        configureAudioSession()

        /* 3. Make call with pjsua_call_make_call().
         *    Then use pjsua's on_call_state() callback to report significant
         *    events in the call's lifecycle, by calling iOS API
         *    CXProvider.reportOutgoingCall(with: startedConnectingAt:) and
         *    CXProvider.reportOutgoingCall(with: ConnectedAt:)
         */
         
        /* 4. If step (3) above returns PJ_SUCCESS, call action.fulfill(),
         *    otherwise call action.fail().
         */
    }

To handle incoming call:

    func provider(_ provider: CXProvider, perform action: CXAnswerCallAction) {
        /* 1. We must not start call audio here, and can only do so
         *    once the audio session has been activated by the system
         *    after having its priority elevated. So, make sure that the sound
         *    device is closed at this point.
         */

        /* 2. Provide your own implementation to configure
         *    the audio session here.
         */
        configureAudioSession()

        /* 3. Answer the call with pjsua_call_answer().
         */

        /* 4. If step (3) above returns PJ_SUCCESS, call action.fulfill(),
         *    otherwise call action.fail().
         */
    }

To start sound device:

    func provider(_ provider: CXProvider, didActivate audioSession: AVAudioSession) {
        /* Start call audio media, now that the audio session has been
         * activated after having its priority boosted.
         *
         * Call pjsua API pjsua_set_snd_dev() here.
         */
    }

API/Symbol Deprecations

1. "kCFStreamNetworkServiceTypeVoIP is deprecated (first deprecated in iOS 9.0 - use PushKit for VoIP control purposes."
   See the PushKit guide above.
   If you want to disable the use of kCFStreamNetworkServiceTypeVoIP, set PJ_IPHONE_OS_HAS_MULTITASKING_SUPPORT to 0.

2. AVFoundation API deprecation:
   "warning: 'devices' is deprecated: first deprecated in iOS 10.0 - Use AVCaptureDeviceDiscoverySession instead."
   Done in r5454.

Reference:

• ​What's new in iOS 10.

For GNU targets:

1. AES GCM crypto requires OpenSSL, so make sure OpenSSL is enabled. Also this requires PJSIP 2.5, as this version has newer libsrtp version. Then make sure that AES GCM is supported by libcrypto by checking the configure output:

   checking for EVP_aes_128_gcm in -lssl... yes
   OpenSSL has AES GCM support, SRTP will use OpenSSL
   

2. Enable the desired AES GCM cryptos in your config_site.h (by default, it's not enabled), for example:

       #define PJMEDIA_SRTP_HAS_AES_GCM_256      1
       #define PJMEDIA_SRTP_HAS_AES_GCM_128      1
   

3. Rebuild PJSIP.

For Windows:

1. Install OpenSSL.

2. In Visual Studio, go to libsrtp project, into folder crypto.

3. Multiple-select cipher/aes_gcm_ossl.c, cipher/aes_icm_ossl.c, hash/hmac_ossl.c, rng/rand_source_ossl.c
   Right-click, then click Properties. Change Configuration to "All Configurations" and Platform to "All Platforms". Go to Configuration Properties - General, then change Excluded From Build from "Yes" to "No".

4. Multiple-select cipher/aes.c cipher/aes_icm.c cipher/aes_cbc.c hash/sha1.c hash/hmac.c rng/rand_source.c rng/prng.c rng/ctr_prng.c
   Right-click, then click Properties. Change Configuration to "All Configurations" and Platform to "All Platforms". Go to Configuration Properties - General, then change Excluded From Build from "No" to "Yes".

5. Click on libsrtp project.
   Right-click, then click Properties. Change Configuration to "All Configurations" and Platform to "All Platforms". Go to Configuration Properties - C/C++ - Preprocessor, then choose Preprocessor Definitions, and add ";OPENSSL".

6. Enable the desired AES GCM cryptos in your config_site.h, for example:

       #define PJMEDIA_SRTP_HAS_AES_GCM_256      1
       #define PJMEDIA_SRTP_HAS_AES_GCM_128      1
   

7. Rebuild libsrtp.

To allow app to set the output window, the video device wrapper needs to support PJMEDIA_VID_DEV_CAP_OUTPUT_WINDOW capability and can be set using pjsua_vid_win_set_win() or VideoWindow::setWindow().

This ticket will add the support for SDL.

Support for the open source implementation of H.264 codec from ​http://www.openh264.org.

Requirement:

• OpenH264 library version 1.6 release (download ​here)
• To avoid conflict, it is recommended to cleanly remove ALL previous/other versions of OpenH264 on your machine, especially if you previously used OpenH264 version 1.0 following the ticket #1758. (If you encounter runtime error openh264.cpp .......SVC encoder Initialize failed, rc=1, it is likely caused by this issue).

Implementation features:

• Constrained Baseline, default level=30 (0x1e)
• packetization modes: 1, 0
• key frame request and indication
• obey remote's fmtp

Notes:

• currently transmission is hard coded to use only single NAL (packetization mode 0) regardless of remote's fmtp for interop reason. The same is also done in ffmpeg wrapper.
• whole frame for file encoding is implemented but currently doesn't work

Below are the steps to use the codec:

For GNU targets:

1. Build the OpenH264 library and install it to a directory (with "make install"). By default it installs to /usr/local. You need to modify the Makefile to install it to different location (=prefix).
2. Run PJSIP's configure. If you specify alternate location (prefix) to OpenH264 installation, specify the location with --with-openh264 option. E.g.:

   ./configure --with-openh264=/home/user/opt
   

3. If you specify an alternate OpenH264 location, add the location path (the path which contains libopenh264*.dylib) to your OS dynamic library path. For example, on Mac:

   export DYLD_LIBRARY_PATH=/Users/teluu/openh264-1.6-mac/lib:$DYLD_LIBRARY_PATH
   

For Visual Studio:

1. Build the OpenH264 library with MinGW make & gcc, and install it to a directory (with "make install"). By default it installs to /usr/local. You need to modify the Makefile to install it to different location (=prefix).
2. Create openh264.lib from openh264.def, note that the static lib libopenh264.a outputted by MinGW gcc cannot be used by VS, so we are going to use the dynamic lib version instead:
   1. open 'Visual Studio command prompt', e.g: Win Start Menu -> MS VS 2005 -> VS Tools -> VS 2005 Command Prompt
   2. change dir to OpenH264 source root directory
   3. run "lib /def:openh264.def", this will create openh264.lib, copy this file to the OpenH264 installation directory
   4. rename or copy libopenh264.dll to openh264.dll and put it in the system path
3. Add OpenH264 include and library installation paths to Visual Studio search path settings.
4. Add this to your config_site.h:

   #define PJMEDIA_HAS_OPENH264_CODEC		1
   

For iOS:

1. Modify the Makefile to install it to a custom folder (find the string PREFIX and modify the value), for example: PREFIX=ios_local. Build the OpenH264 library (example: make OS="ios" ARCH=armv7). Install it to the specified directory (example: make OS="ios" ARCH=armv7 install).
2. Run PJSIP's configure with --with-openh264 option. E.g.:

   ./configure --with-openh264=/Users/teluu/openh264-iphone/ios_local
   

3. Copy libopenh264.a to your XCode project directory. Create a folder ios_local/lib inside your XCode project directory (example: for our ipjsua sample app, the directory will be: (PJPROJECT_DIR)/pjsip-apps/src/pjsua/ios/ios_local/lib). Then copy OpenH264 dynamic library files to the folder. Example:

   cp openh264-iphone/ios_local/lib/libopenh264.a pjproject-iphone/pjsip-apps/src/pjsua/ios/
   cp openh264-iphone/ios_local/lib/*.dylib pjproject-iphone/pjsip-apps/src/pjsua/ios/ios_local/lib
   

4. Add libopenh264.a and iOS-SDK's libstdc++.dylib to your application project in XCode.

For Android:

1. Build the OpenH264 library according to the instructions provided in README.md at the OpenH264 directory. Example for arm64: make OS=android NDKROOT=/home/android-ndk-r13b TARGET=android-15 ARCH=arm64 NDKLEVEL=21 (note that due to the different compiler used (gcc/clang) and system folder structure, using newer NDK may require some changes in the OpenH264 source base). There may be a failure when building the Open264 sample app, but it can be safely ignored since the library itself must have been successfully built by that point.
2. Modify the Makefile to install it to a different location (modify the "prefix") and run "make install". Manually copy libopenh264.so to "prefix"/lib, if necessary.
3. Copy libopenh264.so to your Android application library directory (such as jniLibs/armeabi, or jniLibs/arm64-v8a).

Currently pjsua_vid_win API will return PJ_EINVAL if the video capture preview is native, such as on iOS. This means that application has to manually get the native window handle and use the OS API directly. However, this could be problematic, for example on iOS, the UIView window also contains a preview layer which is not exposed to the application. Hence, resizing the window container itself (by calling the iOS API) will not resize the preview layer itself.

Thus, since the underlying pjmedia videodev implementation already supports such capabilities, this ticket will enchance pjsua_vid_win API to support them as well.

Add callback on_create_media_transport_srtp()/onCreateMediaTransportSrtp() which allows app to modify the SRTP setting in order to specify the cryptos and keys which are going to be used.

WebRTC AEC support is added in ticket #1888. However, building WebRTC separately has a few drawbacks:

• Following the recommended steps to build WebRTC, it is required to download the entire project which is large in size (approximately 6 GB), while we actually only require a small portion of it (namely, the AEC component).
• The latest version of WebRTC has some modifications which is incompatible with our implementation. Thus, we can only use the version circa October 2015 or before.

This ticket will add WebRTC to the third party folder, use a specific version and include only the necessary source.

Below is the steps to build and use WebRTC:

For GNU targets

1. By default, WebRTC will always be built automatically. This automatic build can be disabled via configure script param --disable-webrtc.
2. if you want to use external WebRTC, specify --with-external-webrtc and set header/library search paths via CFLAGS and LDFLAGS environment variables. See ticket #1888 for more details as to which libraries are required.

Additional step for iOS targets

1. Add third-party/lib/libwebrtc-[TARGET].a to application project in XCode.

Supported architectures for Android

1. On Android, we successfully tested with these architectures: armeabi (armv7), armeabi-v7a, arm64-v8a, mips, mips64, x86, x86_64.

For Visual Studio

1. libwebrtc will always be built automatically.
2. To enable WebRTC AEC, add this to your config_site.h:

   #define PJMEDIA_HAS_WEBRTC_AEC		1
   

Add PJMEDIA_AUD_DEV_CAP_INPUT_SOURCE capability (as an alias of PJMEDIA_AUD_DEV_CAP_INPUT_ROUTE) to support changing audio input source. The value of this capability is PJMEDIA_AUD_DEV_ROUTE_CUSTOM + the_desired_audio_input_source_constants.

For example:

// 6 is VOICE_RECOGNITION
unsigned aud_source = PJMEDIA_AUD_DEV_ROUTE_CUSTOM | 6;
pjmedia_aud_param_set_cap(&param, PJMEDIA_AUD_DEV_CAP_INPUT_SOURCE,
			  &aud_source);

Valid audio input source values can be found in ​Android doc here.

Thanks to Johan Lantz for the suggestion.

The ticket will provide the TLS transport information via the existing Endpoint::onTransportState() callback. This way, application can check the SSL certificate information in PJSUA2 and decide to keep the transport or close it by calling Endpoint::transportShutdown().

The following will be implemented in this ticket:

• TlsInfo structure, corresponding to pjsip_tls_state_info and pj_ssl_sock_info
• SslCertInfo structure, corresponding to pj_ssl_cert_info
• field TlsInfo.cipherName, containing the information from pj_ssl_cipher_name() of pj_ssl_sock_info.cipher
• field TlsInfo.verifyMsgs, containing the information from pj_ssl_cert_get_verify_status_strings() of pj_ssl_sock_info.verify_status
• API Endpoint::transportShutdown, corresponding to pjsip_transport_shutdown()

This patch changes the BSD sockets implementation of QoS to use IPV6_TCLASS for IPv6 sockets instead of IP_TOS.

Summary of changes:

• new constant PJ_IPV6_TCLASS, wrapper of IPV6_TCLASS (if defined) or 0xFFFF, because there is no sane default - all operating systems that implement the IPV6_TCLASS option assign a different value to it, and no RFC standardizes it
• pj_IPV6_TCLASS() wrapper for PJ_IPV6_TCLASS constant
• pj_sock_set_qos_params and pj_sock_get_qos_params now check for the socket address family before trying to set or get the socket's QoS: if the socket is IPv4, they will use pj_IP_TOS() as before, if the socket is IPv6 they will use pj_IPV6_TCLASS() instead. Before, using QoS on an IPv6 socket would unconditionally fail with PJ_STATUS_FROM_OS(EINVAL), now it will succeed if the OS supports IPV6_TCLASS (or fail with PJ_STATUS_FROM_OS(ENOPROTOOPT) if it doesn't)

Thanks to Michele Cicciotti for the patch.

The latest XNU kernel, shipped with iOS 10, Mac OS 10.11, AppleTV OS 10, etc. supports a new socket option, SO_NET_SERVICE_TYPE, a high level API for configuring a socket's QoS, which the OS will appropriately and transparently map to a DSCP or WMM priority depending on the network connection.

This patch introduces a new implementation of the socket QoS functions specifically for the Darwin platform, that makes use of SO_NET_SERVICE_TYPE when available, and falls back to IP_TOS/IPV6_TCLASS elsewhere.

Changes introduced by this patch:

• build system and configuration script changes, so that the BSD sockets implementation of QoS is no longer the default on all platforms, but only on non-Darwin platforms. On Darwin platforms, the new Darwin-specific QoS implementation is used instead
• new PJ_QOS_TYPE_SIGNALLING in pj_qos_type which corresponds to NET_SERVICE_TYPE_SIG
• new PJLIB constant OSERR_ENOPROTOOPT, that maps to ENOPROTOOPT (WSAENOPROTOOPT for Winsock)
• new configuration constant PJ_QOS_DARWIN, that can be used as the value of PJ_QOS_IMPLEMENTATION
• new, Darwin-specific implementations of pj_sock_set_qos_params, pj_sock_set_qos_type, pj_sock_get_qos_params and pj_sock_get_qos_type, based on SO_NET_SERVICE_TYPE, and falling back to IP_TOS/IPV6_TCLASS

Backwards/forwards compatibility considerations: on older Darwin platforms, the SDK doesn't define SO_NET_SERVICE_TYPE, and QoS functions always use IP_TOS/IPV6_TCLASS instead. Running code compiled with an old SDK on a new operating system has the same behavior as it had before, and like before with the caveat that, at some point in the future, the OS may start ignoring the value set with IP_TOS/IPV6_TCLASS on newer Darwin platforms, SO_NET_SERVICE_TYPE is always used.

Running code compiled with a new SDK on an old operating system makes attempts to get/set SO_NET_SERVICE_TYPE fail with ENOPROTOOPT, which the QoS function translate to PJ_ENOTSUP, which is handled with a runtime fallback on the old IP_TOS/IPV6_TCLASS code.

For future reference, mapping of pj_qos_type might be changed according to the new standard. Please have a look at the draft: ​https://tools.ietf.org/html/draft-ietf-tsvwg-ieee-802-11-00

Thanks to Michele Cicciotti for the patch.

This feature could be useful for specifying PN (Push Notification) information to the server, such as the one described in the draft ​Push Notifications in the Session Initiation Protocol (SIP)

For example:

Contact: <sip:alice@alicemobile.example.com;
     pn-type=acme;pn-methods="INVITE";
     pn-uri="https://pn.acme.example.com/ZTY4ZDJlMzODE1NmUgKi0K">

This ticket adds the function pjsip_tdata_get_dlg() which is similar to the pjsip_rdata_get_dlg(), but for tdatas. This is very useful to use from inside a module to know if a particular outgoing request/response should be modified based on it belonging to a dialog.

Thanks to Kal (b17 c0de) for the patch.

This ticket will introduce pjmedia_rtp_decode_rtp2() which is similar to pjmedia_rtp_decode_rtp() but adds additional out parameter to return the additional RTP header in the packet if any. Currently the additional RTP header will point to RTP header extension. It would be useful to other people who need to parse the RTP header extension.

Thanks to Kal (b17 c0de) for the original patch.

This ticket will add the following enhancements to TLS:

1. selection of elliptic curve
2. selection of signature algorithm
3. reseeding of random with other entropy sources.

which may be useful to conform to ​NIAP protection profile.

Add support to generate a synthesized IPv6 address from IPv4 address. The ability to synthesize IPv6 addresses was added to getaddrinfo in iOS 9.2 and OS X 10.11.2 (reference: ​Apple's doc).

Thanks to Jamil Abdullayev for the patch.

This patch adds a attach2() member function to pjmedia_transport_op and the corresponding pjmedia_transport_attach2() function.

These new functions take an additional parameter for the media stream and type. This allows all transport adapters in the chain to have access to the media stream and type.

Thanks to Kal (b17 c0de) for the original patch.