wiki:Video_Users_Guide

Context Navigation

Version 48 (modified by bennylp, 13 years ago) (diff)
--

PJSIP Video User's Guide

Table of Contents

Building with Video Support
Building the GUI Sample Application
1. GNU Build System (Mac OS X, Linux, etc)
2. Visual Studio
Using Video API (pjsua-lib)
Video API Reference (pjsua-lib)

Video is available on PJSIP version 2.0 and later. Only desktop platforms are supported, mobile devices such as iOS are not yet supported. This document describes how to use the video feature with PJSIP.

Building with Video Support

Common Requirements for Desktop Platforms

The following components are needed for video:

SDL version 1.3.
- Note that this is the newer and not yet released SDL library, hence most likely it won't be installed by default on your system. You can download the source snapshot or from the Mercurial repository from http://www.libsdl.org/hg.php.

ffmpeg development library. We tested with ffmpeg version ranging from 0.5.1 (from circa 2009) to 0.10.

Note that for H.264 support, you need newer releases (October 2011 onwards), and it needs libz too.

Build with at least:

$ ./configure --enable-shared --disable-static --enable-memalign-hack
                                # add other options if needed, e.g: optimization, install dir, search path 
                                # particularly CFLAGS and LDFLAGS for x264
                                # to enable H264, add "--enable-gpl --enable-libx264"
$ make && make install

Optional for H.264: libx264. We tested with the latest from git (as of October 2011):

$ ./configure --enable-static      # add options if needed, e.g: optimization, install dir, search path
$ make && make install-lib-static  # default install dir is /usr/local

Optional: Qt development SDK for building the video GUI sample. We tested with version 4.6 or later.
- without this you can still enjoy video with pjsua console application

Mac OS X, Linux, *nix (GNU Makefile)

1. Requirements

The following components are needed:

The components in common requirements above.
Mac OS X specific: You might have an old libbz2 for older system, in that case it has to be rebuild (needed by ffmpeg H.264 support)
Linux specific: Video4Linux2 (v4l2) development library is required.

2. Running aconfigure

Video requirements will be detected by the configure script. Pay attention to the following output (the sample below was taken on a Mac):

checking SDL availability..... 1.3.0
checking for deflate in -lz... yes
checking for x264_free in -lx264... yes
checking for avdevice_version in -lavdevice... yes
checking for av_register_all in -lavformat... yes
checking for avcodec_init in -lavcodec... yes
checking for sws_scale in -lswscale... yes
checking for av_malloc in -lavutil... yes
checking for avcore_version in -lavcore... no
checking for v4l2_open in -lv4l2... no

Alternative prefixes for SDL and FFMPEG may be specified with --with-sdl and -with-ffmpeg options respectively. This points to the prefix value that was supplied when building SDL and/or FFMPEG. For example, if libavformat.a is in /usr/local/lib, then the prefix is /usr/local.

3. Configuring video support

Add this to your config_site.h:

#define PJMEDIA_HAS_VIDEO	1
#define PJMEDIA_HAS_FFMPEG	        1
#define PJMEDIA_VIDEO_DEV_HAS_SDL       1
#define PJMEDIA_HAS_FFMPEG_CODEC_H264   1 /* Optional. Note: require libx264 and ffmpeg must be built with h264 enabled */

4. Build the library

As usual with:

make dep && make clean && make

Once the build is successful, you will have pjsua executable containing video support. Additionally, you may want to build the GUI sample application too.

Windows Desktop (Visual Studio)

Visual Studio users will need to configure things manually as explained below.

1. Requirements

Windows requirements:

The components in common requirements above.
Here are sample steps for building SDL, libx264, and ffmpeg:
1. SDL 1.3 comes with VS project settings, e.g: for VS2005 just open VisualC\SDL_VS2005.sln and build.
2. Get MSYS|MinGW for building libx264 and ffmpeg, and also its gcc compiler suite. Note:
  - It is recommended to use gcc 4 or above to build ffmpeg.
  - To avoid problems, put MSYS, libx264, and ffmpeg in folders that do not contain space, e.g: C:\msys, C:\devlib\ffmpeg.
3. libx264 (optional, if H264 is wanted), in MSYS console, as detailed in common requirements.
4. ffmpeg, in MSYS console, as detailed in common requirements.
  - To use ffmpeg with VS, inttypes.h and stdint.h will be needed, check here.

DirectShow SDK, included in Windows SDK. The minimum component required within the SDK is Windows Development Headers and Libraris and Samples. As there are several versions of SDK we found they have different issues:
1. If you don't need Windows 7 features, the recommended SDK is Windows SDK Update for Windows Vista. This is because it doesn't have IP Helper API header bug.
2. If you need Windows 7 features then use Windows SDK for Windows 7 and .NET Framework 4 and follow the instructions on fixing IP Helper API header bug. Also if you are using Visual Studio 2005 then you will need to patch it using MS Knowledge Base 949009

2. Configure directories

Configure Visual Studio directories:

Add include and library paths for the required components:
1. DirectShow SDK
2. SDL
3. ffmpeg
Build Baseclasses sample and add the path to Visual Studio:
1. Look for the Baseclasses project, and add the include path to Visual Studio configuration. Important: the include path for the Baseclasses must be higher than Windows SDK include path.
2. In the property of the Baseclasses project, change the code generation to Multi-threaded Debug, instead of the default Multi-threaded DLL.
3. Compile the baseclasses project in the samples, and add the resulting library to the path of Visual Studio configuration

3. Configure Video Settings

Add these to your config_site.h:

#define PJMEDIA_HAS_VIDEO	        1
#define PJMEDIA_HAS_FFMPEG	        1
#define PJMEDIA_VIDEO_DEV_HAS_SDL       1
#define PJMEDIA_VIDEO_DEV_HAS_DSHOW     1
#define PJMEDIA_HAS_FFMPEG_CODEC_H264   1 /* Optional. Note: require libx264 and ffmpeg must be built with h264 enabled */

4. Rebuild the Solution

Once the build is successful, you will have pjsua executable containing video support. Additionally, you may want to build the GUI sample application too.

Building the GUI Sample Application

We have a GUI sample application with video support. The project is located under pjsip-apps/src/vidgui. It is not built by default, and you need Qt SDK to build it.

GNU Build System (Mac OS X, Linux, etc)

Follow these steps to build vidgui sample:

Go to vidgui source directory:
```
$ cd pjsip-apps/src/vidgui
```
Generate Makefile. For Linux:
```
$ qmake
```
and for Mac OS X:
```
$ qmake -spec macx-g++
```
Build the app:
```
$ make
```

Visual Studio

Follow these steps to build vidgui sample with Visual Studio:

Open command prompt, and
```
cd pjsip-apps\src\vidgui
```
Generate project files:
```
qmake -tp vc
```
Open vidgui.vcproj project.
Save the solution, and build the project

Using Video API (pjsua-lib)

This section provides several sample scenarios of using video in your application. Please see Video API Reference section for a more complete documentation about the Video API.

Enabling Video

By default, video is enabled in pjsua_call_setting, via vid_cnt setting.

Incoming Video Call

Incoming video will be accepted/rejected depending on whether video is enabled in the call setting (see above). If video is enabled, incoming video will be accepted as long as we have matching codec for it. However, this does not necessarily mean that the video will be displayed automatically to the screen, nor that outgoing video will be transmitted automatically, as there will be separate settings for these. Outgoing video behavior will be explained in the next section.

Display Incoming Video Automatically

By default, incoming video is not displayed automatically, since the app may want to seek user approval first. Use the following code to change this behavior on per account basis:

pjsua_acc_config cfg;

pjsua_acc_config_default(&cfg);
cfg.vid_in_auto_show = PJ_TRUE;

Show or Hide Incoming Video

Regardless of the setting above, you can use the following steps to show or hide the display incoming video:

Use pjsua_call_get_vid_stream_idx() or enumerate the call's media stream to find the media index of the default video. If there are multiple video streams in a call, the default video is the first active video media in the call.

Locate the media information of the specified stream index in the pjsua_call_info, and acquire the window ID associated with the remote video. Sample code:

int vid_idx;
pjsua_vid_win_id wid;

vid_idx = pjsua_call_get_vid_stream_idx(call_id);
if (vid_idx >= 0) {
    pjsua_call_info ci;

    pjsua_call_get_info(call_id, &ci);
    wid = ci.media[vid_idx].stream.vid.win_in;
}

Using the video window ID, you may retrieve the associated native video handle with pjsua_vid_win_get_info() and then show or hide the video window using native API, or use pjsua_vid_win_set_show() to show/hide the window using PJSUA API. See Working with Window section below for information on manipulating video windows.

Controlling Incoming Video Stream

Controlling the video window above will not cause any re-INVITE or UPDATE to be sent to remote, since the operation occurs locally. However, if you wish, you may alter the incoming video stream with pjsua_call_set_vid_strm() API, and this will cause re-INVITE or UPDATE to be sent to negotiate the new SDP. The relevant operation to control incoming video with pjsua_call_set_vid_strm() are:

PJSUA_CALL_VID_STRM_CHANGE_DIR: change the media direction (e.g. to "sendonly", or even "inactive")
PJSUA_CALL_VID_STRM_REMOVE: remove the media stream altogether by settings its port to zero
PJSUA_CALL_VID_STRM_ADD: add new video media stream

Since pjsua_call_set_vid_strm() will result in renegotiation of the SDP in a re-INVITE or UPDATE transaction, the result of this operation will not be available immediately. Application can monitor the status by implementing on_call_media_state() callback and enumerate the media stream status with pjsua_call_info.

Incoming Re-offer

If the re-offer contains video, incoming re-offer will be automatically answered with current video setting in the call setting. Currently there is no callback for this, however application can always watch for media update via on_call_media_state() callback.

Outgoing Video Call

Outgoing video is enabled/disabled depending on the call setting. To initiate a call with video in the SDP as inactive, you can disable the video in the call setting and set pjsua_call_setting.flag with PJSUA_CALL_INCLUDE_DISABLED_MEDIA.

Outgoing Video Transmission

Outgoing video transmission is independent from the incoming video transmission; each can be operated separately. Note that outgoing video transmission is not started by default, not even when incoming offer contains video support. This behavior is controlled by pjsua_acc_config.vid_out_auto_transmit setting, which default to PJ_FALSE. Setting this to PJ_TRUE will cause video transmission to be started automatically on each outgoing calls and on incoming calls that indicates video support in its offer. However, it is more flexible and appropriate to leave this setting at PJ_FALSE, and add video later during the call by using pjsua_call_set_vid_strm() API, as will be explained shortly.

Default Capture Device

The default capture device that is used by an account is configured in pjsua_acc_config.vid_cap_dev setting. It is more convenient to set the "correct" device here rather than having to set it in every other API calls later.

Controlling Video Stream

Application uses pjsua_call_set_vid_strm() API to control video stream on a call. The following op_code can be used on the API to control the video media:

PJSUA_CALL_VID_STRM_ADD: add a new video stream
PJSUA_CALL_VID_STRM_REMOVE: remove video stream (set port to zero)
PJSUA_CALL_VID_STRM_CHANGE_DIR: change direction or deactivate (i.e. set direction to "inactive")
PJSUA_CALL_VID_STRM_CHANGE_CAP_DEV: change capture device
PJSUA_CALL_VID_STRM_START_TRANSMIT: start previously stopped transmission
PJSUA_CALL_VID_STRM_STOP_TRANSMIT: stop transmission

Some of the video operations above require re-INVITE or UPDATE to be sent, hence the result will not be available immediately. In that case, application can implement on_call_media_state() callback and inspect the resulting negotiation by looking at the pjsua_call_info. Please see Video Call Manipulation in the API reference section below for more information about the operations above.

Add or Remove Video

You can set the field vid_cnt of pjsua_call_setting to the desired video count to add/remove video, then send the reinvite/update. Alternatively, you can use pjsua_call_set_vid_strm() API to control the video stream on a call (see Controlling Incoming Video Stream or Controlling Video Stream above).

Working with Video Window

Video Window represents all window objects on the screen that the library creates. The video window can display incoming video, preview, and/or other video playbacks.

Application may retrieve video windows from the following places:

for calls, the video window of incoming video stream is contained in the media stream inside pjsua_call_info structure.
preview window associated with a capture device can be queried with pjsua_vid_preview_get_win().
for all other purposes, application may enumerate all video windows with pjsua_vid_enum_wins().

Application retrieves pjsua_vid_win_info with pjsua_vid_win_get_info(). The one window property that most applications will be interested with is the native window handle of the video. The native video handle is contained by pjmedia_vid_dev_hwnd structure inside pjsua_vid_win_info. Application can use the native handle to embed the video window into application's GUI structure. Alternatively, the library also provides few simple and most commonly used API to operate the window, such as pjsua_vid_win_set_show(), pjsua_vid_win_set_size(), etc., however the availability of these APIs are not guaranteed since it depends on the underlying backend device.

Modifying video codec parameters for video call

Video codec parameters are specified in pjmedia_vid_codec_param. The codec parameters provide separate settings for each direction, encoding and decoding. Any modifications on video codec parameters can be applied using pjsua_vid_codec_set_param(), here is a sample code for reference:

const pj_str_t codec_id = {"H264", 4};
pjmedia_vid_codec_param param;

pjsua_vid_codec_get_param(&codec_id, &param);

/* Modify param here */
...

pjsua_vid_codec_set_param(&codec_id, &param);

Size or resolution

Specify video picture dimension.

For encoding direction, configured via pjmedia_vid_codec_param.enc_fmt.det.vid.size, e.g:
```
/* Sending 1280 x 720 */
param.enc_fmt.det.vid.size.w = 1280;
param.enc_fmt.det.vid.size.h = 720;
```
Note that there is a possibility that the value will be adjusted to follow remote capability. For example, if remote signals that maximum resolution supported is 640 x 480 and locally the encoding direction size is set to 1280 x 720, then 640 x 480 will be used.

For decoding direction, two steps are needed:

pjmedia_vid_codec_param.dec_fmt.det.vid.size should be set to the highest value expected for incoming video size.

signalling to remote, configured via codec specific SDP format parameter (fmtp): pjmedia_vid_codec_param.dec_fmtp.

H263-1998, e.g:

/* 1st preference: 352 x 288 (CIF) */
param.dec_fmtp.param[n].name = pj_str("CIF");
/* The value actually specifies framerate, see framerate section below */
param.dec_fmtp.param[n].val = pj_str("1");
/* 2nd preference: 176 x 144 (QCIF) */
param.dec_fmtp.param[n+1].name = pj_str("QCIF");
/* The value actually specifies framerate, see framerate section below */
param.dec_fmtp.param[n+1].val = pj_str("1");

H264, the size is implicitly specified in H264 level (check the standard specification or this) and on SDP, the H264 level is signalled via H264 SDP fmtp profile-level-id, e.g:

/* Can receive up to 1280×720 @30fps */
param.dec_fmtp.param[n].name = pj_str("profile-level-id");
/* Set the profile level to "1f", which means level 3.1 */
param.dec_fmtp.param[n].val = pj_str("xxxx1f");

Framerate

Specify number of frames processed per second.

For encoding direction, configured via pjmedia_vid_codec_param.enc_fmt.det.vid.fps, e.g:
```
/* Sending @30fps */
param.enc_fmt.det.vid.fps.num   = 30;
param.enc_fmt.det.vid.fps.denum = 1;
```
Note that there is a possibility that the value will be adjusted to follow remote capability. For example, if remote signals that maximum framerate supported is 10fps and locally the encoding direction framerate is set to 30fps, then 10fps will be used.
For decoding direction, two steps are needed:
1. pjmedia_vid_codec_param.dec_fmt.det.vid.fps should be set to the highest value expected for incoming video framerate.
2. signalling to remote, configured via codec specific SDP format parameter (fmtp): pjmedia_vid_codec_param.dec_fmtp.
  - H263-1998, maximum framerate is specified per size/resolution basis, check here for more info.
```
/* 3000/(1.001*2) fps for CIF */
param.dec_fmtp.param[m].name = pj_str("CIF");
param.dec_fmtp.param[m].val = pj_str("2");
/* 3000/(1.001*1) fps for QCIF */
param.dec_fmtp.param[n].name = pj_str("QCIF");
param.dec_fmtp.param[n].val = pj_str("1");
```
  - H264, similar to size/resolution, the framerate is implicitly specified in H264 level (check the standard specification or this) and the H264 level is signalled via H264 SDP fmtp profile-level-id, e.g:
```
/* Can receive up to 1280×720 @30fps */
param.dec_fmtp.param[n].name = pj_str("profile-level-id");
param.dec_fmtp.param[n].val = pj_str("xxxx1f");
```

Bitrate

Specify bandwidth requirement for video payloads stream delivery.

This is configurable via pjmedia_vid_codec_param.enc_fmt.det.vid.avg_bps and pjmedia_vid_codec_param.enc_fmt.det.vid.max_bps, e.g:

/* Bitrate range preferred: 512-1024kbps */
param.enc_fmt.det.vid.avg_bps = 512000;
param.enc_fmt.det.vid.max_bps = 1024000;

Notes:

This setting is applicable for encoding and decoding direction, currently there is no way to set asymmetric bitrate. By decoding direction, actually it just means that this setting will be queried when generating bandwidth info for local SDP (see next point).
The bitrate setting of all codecs will be enumerated and the highest value will be signalled in bandwidth info in local SDP (see ticket #1244).
There is a possibility that the encoding bitrate will be adjusted to follow remote bitrate setting, i.e: read from SDP bandwidth info (b=TIAS line) in remote SDP. For example, if remote signals that maximum bitrate is 128kbps and locally the bitrate is set to 512kbps, then 128kbps will be used.

If codec specific bitrate setting signalling (via SDP fmtp) is desired, e.g: MaxBR for H263, application should put the SDP fmtp manually, for example:

/* H263 specific maximum bitrate 512kbps */
param.dec_fmtp.param[n].name = pj_str("MaxBR");
param.dec_fmtp.param[n].val = pj_str("5120"); /* = max_bps / 100 */

Video API Reference (pjsua-lib)

This section explains and lists the Video API as it was available when this document is written. For a richer and more up to date list, please see Video API reference doxygen documentation.

The Video API is classified into the following categories.

Device enumeration API

The following API is available:

/**
 * Get the number of video devices installed in the system.
 *
 * @return		The number of devices.
 */
PJ_DECL(unsigned) pjsua_vid_dev_count(void);

/**
 * Retrieve the video device info for the specified device index.
 *
 * @param id		The device index.
 * @param vdi		Device info to be initialized.
 *
 * @return		PJ_SUCCESS on success, or the appropriate error code.
 */
PJ_DECL(pj_status_t) pjsua_vid_dev_get_info(pjmedia_vid_dev_index id,
                                            pjmedia_vid_dev_info *vdi);

/**
 * Enum all video devices installed in the system.
 *
 * @param info		Array of info to be initialized.
 * @param count		On input, specifies max elements in the array.
 *			On return, it contains actual number of elements
 *			that have been initialized.
 *
 * @return		PJ_SUCCESS on success, or the appropriate error code.
 */
PJ_DECL(pj_status_t) pjsua_vid_enum_devs(pjmedia_vid_dev_info info[],
					 unsigned *count);

In addition, the PJMEDIA videodev also provides this API to detect change in device availability:

/**
 * Refresh the list of video devices installed in the system. This function
 * will only refresh the list of videoo device so all active video streams will
 * be unaffected. After refreshing the device list, application MUST make sure
 * to update all index references to video devices (i.e. all variables of type
 * pjmedia_vid_dev_index) before calling any function that accepts video device
 * index as its parameter.
 *
 * @return		PJ_SUCCESS on successful operation or the appropriate
 *			error code.
 */
PJ_DECL(pj_status_t) pjmedia_vid_dev_refresh(void);

Video preview API

The video preview API can be used to show the output of capture device to a video window:

/**
 * Parameters for starting video preview with pjsua_vid_preview_start().
 * Application should initialize this structure with
 * pjsua_vid_preview_param_default().
 */
typedef struct pjsua_vid_preview_param
{
    /**
     * Device ID for the video renderer to be used for rendering the
     * capture stream for preview.
     */
    pjmedia_vid_dev_index	rend_id;
} pjsua_vid_preview_param;


/**
 * Start video preview window for the specified capture device.
 *
 * @param id		The capture device ID where its preview will be
 * 			started.
 * @param prm		Optional video preview parameters. Specify NULL
 * 			to use default values.
 *
 * @return		PJ_SUCCESS on success, or the appropriate error code.
 */
PJ_DECL(pj_status_t) pjsua_vid_preview_start(pjmedia_vid_dev_index id,
                                             pjsua_vid_preview_param *prm);

/**
 * Get the preview window handle associated with the capture device, if any.
 *
 * @param id		The capture device ID.
 *
 * @return		The window ID of the preview window for the
 * 			specified capture device ID, or NULL if preview
 * 			does not exist.
 */
PJ_DECL(pjsua_vid_win_id) pjsua_vid_preview_get_win(pjmedia_vid_dev_index id);

/**
 * Stop video preview.
 *
 * @param id		The capture device ID.
 *
 * @return		PJ_SUCCESS on success, or the appropriate error code.
 */
PJ_DECL(pj_status_t) pjsua_vid_preview_stop(pjmedia_vid_dev_index id);

Video Configuration

Video is enabled/disabled on pjsua_call_setting:

    /**
     * Number of simultaneous active video streams for this call. Setting
     * this to zero will disable video in this call.
     *
     * Default: 1 (if video feature is enabled, otherwise it is zero)
     */
    unsigned         vid_cnt;

Video settings are mostly configured on pjsua_acc_config with the following fields:

    /**
     * Specify whether incoming video should be shown to screen by default.
     * This applies to incoming call (INVITE), incoming re-INVITE, and
     * incoming UPDATE requests.
     *
     * Regardless of this setting, application can detect incoming video
     * by implementing \a on_call_media_state() callback and enumerating
     * the media stream(s) with #pjsua_call_get_info(). Once incoming
     * video is recognised, application may retrieve the window associated
     * with the incoming video and show or hide it with
     * #pjsua_vid_win_set_show().
     *
     * Default: PJ_FALSE
     */
    pj_bool_t        vid_in_auto_show;

    /**
     * Specify whether outgoing video should be activated by default when
     * making outgoing calls and/or when incoming video is detected. This
     * applies to incoming and outgoing calls, incoming re-INVITE, and
     * incoming UPDATE. If the setting is non-zero, outgoing video
     * transmission will be started as soon as response to these requests
     * is sent (or received).
     *
     * Regardless of the value of this setting, application can start and
     * stop outgoing video transmission with #pjsua_call_set_vid_strm().
     *
     * Default: PJ_FALSE
     */
    pj_bool_t        vid_out_auto_transmit;

    /**
     * Specify the default capture device to be used by this account. If
     * \a vid_out_auto_transmit is enabled, this device will be used for
     * capturing video.
     *
     * Default: PJMEDIA_VID_DEFAULT_CAPTURE_DEV
     */
    pjmedia_vid_dev_index vid_cap_dev;

    /**
     * Specify the default rendering device to be used by this account.
     *
     * Default: PJMEDIA_VID_DEFAULT_RENDER_DEV
     */
    pjmedia_vid_dev_index vid_rend_dev;

Video Call Manipulation

The default video behavior for a call is controlled by the account settings above. On top of that, the application can manipulate video of an already-going call by using pjsua_call_set_vid_strm() API:

/**
 * This enumeration represents video stream operation on a call.
 * See also #pjsua_call_vid_strm_op_param for further info.
 */
typedef enum pjsua_call_vid_strm_op
{
    /**
     * No operation
     */
    PJSUA_CALL_VID_STRM_NO_OP,

    /**
     * Add a new video stream. This will add a new m=video line to
     * the media, regardless of whether existing video is/are present
     * or not.  This will cause re-INVITE or UPDATE to be sent to remote
     * party.
     */
    PJSUA_CALL_VID_STRM_ADD,

    /**
     * Remove/disable an existing video stream. This will
     * cause re-INVITE or UPDATE to be sent to remote party.
     */
    PJSUA_CALL_VID_STRM_REMOVE,

    /**
     * Change direction of a video stream. This operation can be used
     * to activate or deactivate an existing video media. This will
     * cause re-INVITE or UPDATE to be sent to remote party.
     */
    PJSUA_CALL_VID_STRM_CHANGE_DIR,

    /**
     * Change capture device of a video stream.  This will not send
     * re-INVITE or UPDATE to remote party.
     */
    PJSUA_CALL_VID_STRM_CHANGE_CAP_DEV,

    /**
     * Start transmitting video stream. This will cause previously
     * stopped stream to start transmitting again. Note that no
     * re-INVITE/UPDATE is to be transmitted to remote since this
     * operation only operates on local stream.
     */
    PJSUA_CALL_VID_STRM_START_TRANSMIT,

    /**
     * Stop transmitting video stream. This will cause the stream to
     * be paused in TX direction, causing it to stop sending any video
     * packets. No re-INVITE/UPDATE is to be transmitted to remote
     * with this operation.
     */
    PJSUA_CALL_VID_STRM_STOP_TRANSMIT,

    /**
     * Send keyframe in the video stream. This will force the stream to
     * generate and send video keyframe as soon as possible. No
     * re-INVITE/UPDATE is to be transmitted to remote with this operation.
     */
    PJSUA_CALL_VID_STRM_SEND_KEYFRAME

} pjsua_call_vid_strm_op;


/**
 * Parameters for video stream operation on a call.
 */
typedef struct pjsua_call_vid_strm_op_param
{
    /**
     * Specify the media stream index. This can be set to -1 to denote
     * the default video stream in the call, which is the first active
     * video stream or any first video stream if none is active.
     *
     * This field is valid for all video stream operations, except
     * PJSUA_CALL_VID_STRM_ADD.
     *
     * Default: -1 (first active video stream, or any first video stream
     *              if none is active)
     */
    int med_idx;

    /**
     * Specify the media stream direction.
     *
     * This field is valid for the following video stream operations:
     * PJSUA_CALL_VID_STRM_ADD and PJSUA_CALL_VID_STRM_CHANGE_DIR.
     *
     * Default: PJMEDIA_DIR_ENCODING_DECODING
     */
    pjmedia_dir dir;

    /**
     * Specify the video capture device ID. This can be set to
     * PJMEDIA_VID_DEFAULT_CAPTURE_DEV to specify the default capture
     * device as configured in the account.
     *
     * This field is valid for the following video stream operations:
     * PJSUA_CALL_VID_STRM_ADD and PJSUA_CALL_VID_STRM_CHANGE_CAP_DEV.
     *
     * Default: capture device configured in account.
     */
    pjmedia_vid_dev_index cap_dev;

} pjsua_call_vid_strm_op_param;



/**
 * Add, remove, modify, and/or manipulate video media stream for the
 * specified call. This may trigger a re-INVITE or UPDATE to be sent
 * for the call.
 *
 * @param call_id	Call identification.
 * @param op		The video stream operation to be performed,
 *			possible values are #pjsua_call_vid_strm_op.
 * @param param		The parameters for the video stream operation,
 *			or NULL for the default parameter values
 *			(see #pjsua_call_vid_strm_op_param).
 *
 * @return		PJ_SUCCESS on success or the appropriate error.
 */
PJ_DECL(pj_status_t) pjsua_call_set_vid_strm (
				pjsua_call_id call_id,
				pjsua_call_vid_strm_op op,
				const pjsua_call_vid_strm_op_param *param);

/**
 * Get the media stream index of the default video stream in the call.
 * Typically this will just retrieve the stream index of the first
 * activated video stream in the call.
 *
 * @param call_id	Call identification.
 *
 * @return		The media stream index or -1 if no video stream
 * 			is present in the call.
 */
PJ_DECL(int) pjsua_call_get_vid_stream_idx(pjsua_call_id call_id);

Video Call Information

New fields have been added to pjsua_call_info to report media information in more detail:

Snippet of pjsua_call_info new fields:

    /** Number of media streams in this call */
    unsigned		media_cnt;

    /** Array of media stream information */
    struct
    {
	/** Media index in SDP. */
	unsigned		index;

	/** Media type. */
	pjmedia_type		type;

	/** Media direction. */
	pjmedia_dir		dir;

	/** Call media status. */
	pjsua_call_media_status status;

	/** The specific media stream info. */
	union {
	    /** Audio stream */
	    struct {
		/** The conference port number for the call.  */
		pjsua_conf_port_id   conf_slot;
	    } aud;

	    /** Video stream */
	    struct {
		/**
		 * The window id for incoming video, if any, or
		 * PJSUA_INVALID_ID.
		 */
		pjsua_vid_win_id     win_in;

		/** The video capture device for outgoing transmission,
		 *  if any, or PJMEDIA_VID_INVALID_DEV
		 */
		pjmedia_vid_dev_index	cap_dev;

	    } vid;
	} stream;

    } media[PJMEDIA_MAX_SDP_MEDIA];

Video Call Stream Information and Statistic

New API has been added to query call's stream information and statistic. Subsequently, the old pjsua_call_get_media_session() API has been deprecated since its use is unsafe.

New API:

/**
 * Media stream info.
 */
typedef struct pjsua_stream_info
{
    /** Media type of this stream. */
    pjmedia_type type;

    /** Stream info (union). */
    union {
	/** Audio stream info */
	pjmedia_stream_info	aud;

	/** Video stream info */
	pjmedia_vid_stream_info	vid;
    } info;

} pjsua_stream_info;


/**
 * Media stream statistic.
 */
typedef struct pjsua_stream_stat
{
    /** RTCP statistic. */
    pjmedia_rtcp_stat	rtcp;

    /** Jitter buffer statistic. */
    pjmedia_jb_state	jbuf;

} pjsua_stream_stat;

/**
 * Get media stream info for the specified media index.
 *
 * @param call_id	The call identification.
 * @param med_idx	Media stream index.
 * @param psi		To be filled with the stream info.
 *
 * @return		PJ_SUCCESS on success or the appropriate error.
 */
PJ_DECL(pj_status_t) pjsua_call_get_stream_info(pjsua_call_id call_id,
                                                unsigned med_idx,
                                                pjsua_stream_info *psi);

/**
 *  Get media stream statistic for the specified media index.
 *
 * @param call_id	The call identification.
 * @param med_idx	Media stream index.
 * @param psi		To be filled with the stream statistic.
 *
 * @return		PJ_SUCCESS on success or the appropriate error.
 */
PJ_DECL(pj_status_t) pjsua_call_get_stream_stat(pjsua_call_id call_id,
                                                unsigned med_idx,
                                                pjsua_stream_stat *stat);

/**
 * Get media transport info for the specified media index.
 *
 * @param call_id	The call identification.
 * @param med_idx	Media stream index.
 * @param t		To be filled with the transport info.
 *
 * @return		PJ_SUCCESS on success or the appropriate error.
 */
PJ_DECL(pj_status_t) 
pjsua_call_get_med_transport_info(pjsua_call_id call_id,
                                  unsigned med_idx,
                                  pjmedia_transport_info *t);

Video Window API

A video window is a rectangular area in your monitor to display video content. The video content may come from remote stream, local camera (in case of preview), AVI playback, or any other video playback. Application mostly will be interested in the native handle of the video window so that it can embed it in its application window, however we also provide simple and commonly used API for manipulating the window.

/**
 * This structure describes video window info.
 */
typedef struct pjsua_vid_win_info
{
    /**
     * Renderer device ID.
     */
    pjmedia_vid_dev_index rdr_dev;

    /**
     * Native window handle.
     */
    pjmedia_vid_dev_hwnd hwnd;

    /**
     * Window show status. The window is hidden if false.
     */
    pj_bool_t	show;

    /**
     * Window position.
     */
    pjmedia_coord pos;

    /**
     * Window size.
     */
    pjmedia_rect_size size;

} pjsua_vid_win_info;


/**
 * Enumerates all video windows.
 *
 * @param id		Array of window ID to be initialized.
 * @param count		On input, specifies max elements in the array.
 *			On return, it contains actual number of elements
 *			that have been initialized.
 *
 * @return		PJ_SUCCESS on success, or the appropriate error code.
 */
PJ_DECL(pj_status_t) pjsua_vid_enum_wins(pjsua_vid_win_id wids[],
					 unsigned *count);


/**
 * Get window info.
 *
 * @param wid		The video window ID.
 * @param wi		The video window info to be initialized.
 *
 * @return		PJ_SUCCESS on success, or the appropriate error code.
 */
PJ_DECL(pj_status_t) pjsua_vid_win_get_info(pjsua_vid_win_id wid,
                                            pjsua_vid_win_info *wi);

/**
 * Show or hide window.
 *
 * @param wid		The video window ID.
 * @param show		Set to PJ_TRUE to show the window, PJ_FALSE to
 * 			hide the window.
 *
 * @return		PJ_SUCCESS on success, or the appropriate error code.
 */
PJ_DECL(pj_status_t) pjsua_vid_win_set_show(pjsua_vid_win_id wid,
                                            pj_bool_t show);

/**
 * Set video window position.
 *
 * @param wid		The video window ID.
 * @param pos		The window position.
 *
 * @return		PJ_SUCCESS on success, or the appropriate error code.
 */
PJ_DECL(pj_status_t) pjsua_vid_win_set_pos(pjsua_vid_win_id wid,
                                           const pjmedia_coord *pos);

/**
 * Resize window.
 *
 * @param wid		The video window ID.
 * @param size		The new window size.
 *
 * @return		PJ_SUCCESS on success, or the appropriate error code.
 */
PJ_DECL(pj_status_t) pjsua_vid_win_set_size(pjsua_vid_win_id wid,
                                            const pjmedia_rect_size *size);

Video Codec API

API for managing video codecs:

/**
 * Enum all supported video codecs in the system.
 *
 * @param id		Array of ID to be initialized.
 * @param count		On input, specifies max elements in the array.
 *			On return, it contains actual number of elements
 *			that have been initialized.
 *
 * @return		PJ_SUCCESS on success, or the appropriate error code.
 */
PJ_DECL(pj_status_t) pjsua_vid_enum_codecs( pjsua_codec_info id[],
					    unsigned *count );


/**
 * Change video codec priority.
 *
 * @param codec_id	Codec ID, which is a string that uniquely identify
 *			the codec (such as "H263/90000"). Please see pjsua
 *			manual or pjmedia codec reference for details.
 * @param priority	Codec priority, 0-255, where zero means to disable
 *			the codec.
 *
 * @return		PJ_SUCCESS on success, or the appropriate error code.
 */
PJ_DECL(pj_status_t) pjsua_vid_codec_set_priority( const pj_str_t *codec_id,
						   pj_uint8_t priority );


/**
 * Get video codec parameters.
 *
 * @param codec_id	Codec ID.
 * @param param		Structure to receive video codec parameters.
 *
 * @return		PJ_SUCCESS on success, or the appropriate error code.
 */
PJ_DECL(pj_status_t) pjsua_vid_codec_get_param(
					const pj_str_t *codec_id,
					pjmedia_vid_codec_param *param);


/**
 * Set video codec parameters.
 *
 * @param codec_id	Codec ID.
 * @param param		Codec parameter to set. Set to NULL to reset
 *			codec parameter to library default settings.
 *
 * @return		PJ_SUCCESS on success, or the appropriate error code.
 */
PJ_DECL(pj_status_t) pjsua_vid_codec_set_param( 
					const pj_str_t *codec_id,
					const pjmedia_vid_codec_param *param);

Context Navigation

PJSIP Video User's Guide

Building with Video Support

Common Requirements for Desktop Platforms

Mac OS X, Linux, *nix (GNU Makefile)

1. Requirements

2. Running aconfigure

3. Configuring video support

4. Build the library

Windows Desktop (Visual Studio)

1. Requirements

2. Configure directories

3. Configure Video Settings

4. Rebuild the Solution

Building the GUI Sample Application

GNU Build System (Mac OS X, Linux, etc)

Visual Studio

Using Video API (pjsua-lib)

Enabling Video

Incoming Video Call

Display Incoming Video Automatically

Show or Hide Incoming Video

Controlling Incoming Video Stream

Incoming Re-offer

Outgoing Video Call

Outgoing Video Transmission

Default Capture Device

Controlling Video Stream

Add or Remove Video

Working with Video Window

Modifying video codec parameters for video call

Size or resolution

Framerate

Bitrate

Video API Reference (pjsua-lib)

Device enumeration API

Video preview API

Video Configuration

Video Call Manipulation

Video Call Information

Video Call Stream Information and Statistic

Video Window API

Video Codec API

Download in other formats: