{{{ #!html
}}} = PJSIP Video User's Guide = '''Table of Contents''' [[PageOutline(2-3,,inline)]] Video is available on PJSIP version 2.0 and later. This document describes how to use the video feature with PJSIP. [[BR]] == Requirements == All platforms: 1. [http://www.libsdl.org/ SDL] '''version 1.3'''. - Note that this is the newer and not yet released SDL library, hence most likely it won't be installed by default on your system. You can download the source snapshot or from the Mercurial repository from http://www.libsdl.org/hg.php. 1. [http://ffmpeg.org/ ffmpeg] development library. 1. Optional: [http://qt.nokia.com/downloads/ Qt development SDK] for building the [#vidgui video GUI sample]. - Linux: 1. Video4Linux2 (v4l2) development library Windows: 1. ? Mac OS X: - Mobile platforms (iPhone, Symbian, Windows Mobile, etc): - Video is not supported on this release yet! [[BR]] == Building with Video Support == === Building with GNU Build System (for Mac OS X, Linux, etc.) === ==== 1. Running aconfigure ==== Video requirements will be detected by the {{{configure}}} script. Pay attention to the following output (the sample below was taken on a Mac): {{{ checking SDL availability..... 1.3 checking for avdevice_version in -lavdevice... yes checking for av_register_all in -lavformat... yes checking for avcodec_init in -lavcodec... yes checking for sws_scale in -lswscale... yes checking for av_malloc in -lavutil... yes checking for avcore_version in -lavcore... yes checking for v4l2_open in -lv4l2... no }}} Once again, we need SDL version 1.3 for the video to work. ==== 2. Configuring video support ==== Add this in your {{{config_site.h}}}: {{{ #define PJMEDIA_HAS_VIDEO 1 }}} ==== 3. Build the library ==== As usual with: {{{ make dep && make clean && make }}} Once the build is successful, you will have '''pjsua''' executable containing video support. Additionally, you may want to build the [#vidgui GUI sample application] too. [[BR]] === Building with Visual Studio === Visual Studio users will need to configure things manually as explained below. ==== 1. Configure directories ==== Add include and library paths for the required components: 1. SDL 1. ffmpeg 1. !DirectX (?) 1. Base classes (?) ==== 2. Configure Video Settings ==== Add these to your {{{config_site.h}}}: {{{ #define PJMEDIA_HAS_VIDEO 1 #define PJMEDIA_HAS_FFMPEG 1 #define PJMEDIA_VIDEO_DEV_HAS_SDL 1 }}} ==== 3. Rebuild the Solution ==== Once the build is successful, you will have '''pjsua''' executable containing video support. Additionally, you may want to build the [#vidgui GUI sample application] too. [[BR]] === GUI Sample Application === #vidgui We have a GUI sample application with video support. The project is located under {{{pjsip-apps/src/vidgui}}}. It is not built by default, and you need [http://qt.nokia.com/downloads/ Qt SDK] to build it. ==== GNU Build System (Mac OS X, Linux, etc) ==== Follow these steps to build vidgui sample: {{{ $ cd pjsip-apps/src/vidgui $ make }}} ==== Visual Studio ==== Follow these steps to build vidgui sample with Visual Studio: 1. Open command prompt, and {{{ cd pjsip-apps\src\vidgui }}} 1. Generate project files: {{{ qmake -tp vc }}} 1. Open vidgui.vcproj project. 1. Save the solution, and build the project [[BR]] == Using Video API (pjsua-lib) == This section provides several sample scenarios of using video in your application. Please see [#vidref Video API Reference] section for a more complete documentation about the Video API. === Enabling video === By default, video is enabled in {{{pjsua_acc_config}}}, via {{{max_video_cnt}}} setting. === Incoming video display === As long as video is enabled in the account config (see above), incoming video by default will be accepted as long as we have matching codec for it. However, this does not necessarily mean that the video will be displayed automatically to the screen, nor that we will automatically transmit outgoing video because of that, as there will be additional settings to control these. Outgoing video behavior will be explained in the following section. By default, incoming video '''is not''' displayed automatically, since the app may want to seek user approval first. Use the following code to change this behavior on per account basis: {{{ pjsua_acc_config cfg; pjsua_acc_config_default(&cfg); cfg.wid_in_auto_show = PJ_TRUE; }}} Regardless of the setting above, you can use the following steps to show or hide the display incoming video: 1. Use {{{pjsua_call_get_vid_stream_idx()}}} or enumerate the call's media stream to find the media index of the default video. If there are multiple video streams in a call, the default video is the first active video media in the call. 1. Locate the media information of the specified stream index in the {{{pjsua_call_info}}}, and acquire the window ID associated with the remote video. Sample code: {{{ int vid_idx; pjsua_vid_win_id wid; vid_idx = pjsua_call_get_vid_stream_idx(call_id); if (vid_idx >= 0) { pjsua_call_info ci; pjsua_call_get_info(call_id, &ci); wid = ci.media[vid_idx].stream.vid.win_in; } }}} 1. Using the video window ID, you may retrieve the associated native video handle with {{{pjsua_vid_win_get_info()}}} and then show or hide the video window using native API, or use {{{pjsua_vid_win_set_show()}}} to show/hide the window using PJSUA API. Controlling the video window above will not cause any re-INVITE or UPDATE to be sent to remote, since the operation occurs locally. However, if you wish, you may alter the incoming video stream with {{{pjsua_call_set_vid_strm()}}} API, and this '''will''' cause re-INVITE or UPDATE to be sent to negotiate the new SDP. The relevant operation to control incoming video with {{{pjsua_call_set_vid_strm()}}} are: - PJSUA_CALL_VID_STRM_CHANGE_DIR: change the media direction (e.g. to "sendonly", or even "inactive") - PJSUA_CALL_VID_STRM_REMOVE: remove the media stream altogether by settings its port to zero - PJSUA_CALL_VID_STRM_ADD: add new video media stream Since {{{pjsua_call_set_vid_strm()}}} will result in renegotiation of the SDP in a re-INVITE or UPDATE transaction, the result of this operation will not be available immediately. Application can monitor the status by implementing '''on_call_media_state()''' callback and enumerate the media stream status with '''pjsua_call_info'''. [[BR]] == Video API Reference (pjsua-lib) == #vidref This section explains and lists the Video API as it was available when this document is written. For a richer and more up to date list, please see [http://www.pjsip.org/docs/latest-2/pjsip/docs/html/group__PJSUA__LIB__VIDEO.htm Video API reference doxygen documentation]. The Video API is classified into the following categories. === Device enumeration API === The following API is available: {{{ /** * Get the number of video devices installed in the system. * * @return The number of devices. */ PJ_DECL(unsigned) pjsua_vid_dev_count(void); /** * Retrieve the video device info for the specified device index. * * @param id The device index. * @param vdi Device info to be initialized. * * @return PJ_SUCCESS on success, or the appropriate error code. */ PJ_DECL(pj_status_t) pjsua_vid_dev_get_info(pjmedia_vid_dev_index id, pjmedia_vid_dev_info *vdi); /** * Enum all video devices installed in the system. * * @param info Array of info to be initialized. * @param count On input, specifies max elements in the array. * On return, it contains actual number of elements * that have been initialized. * * @return PJ_SUCCESS on success, or the appropriate error code. */ PJ_DECL(pj_status_t) pjsua_vid_enum_devs(pjmedia_vid_dev_info info[], unsigned *count); }}} In addition, the [http://www.pjsip.org/docs/latest-2/pjmedia/docs/html/group__video__device__reference.htm PJMEDIA videodev] also provides this API to detect change in device availability: {{{ /** * Refresh the list of video devices installed in the system. This function * will only refresh the list of videoo device so all active video streams will * be unaffected. After refreshing the device list, application MUST make sure * to update all index references to video devices (i.e. all variables of type * pjmedia_vid_dev_index) before calling any function that accepts video device * index as its parameter. * * @return PJ_SUCCESS on successful operation or the appropriate * error code. */ PJ_DECL(pj_status_t) pjmedia_vid_dev_refresh(void); }}} === Video preview API === The video preview API can be used to show the output of capture device to a video window: {{{ /** * Parameters for starting video preview with pjsua_vid_preview_start(). * Application should initialize this structure with * pjsua_vid_preview_param_default(). */ typedef struct pjsua_vid_preview_param { /** * Device ID for the video renderer to be used for rendering the * capture stream for preview. */ pjmedia_vid_dev_index rend_id; } pjsua_vid_preview_param; /** * Start video preview window for the specified capture device. * * @param id The capture device ID where its preview will be * started. * @param prm Optional video preview parameters. Specify NULL * to use default values. * * @return PJ_SUCCESS on success, or the appropriate error code. */ PJ_DECL(pj_status_t) pjsua_vid_preview_start(pjmedia_vid_dev_index id, pjsua_vid_preview_param *prm); /** * Get the preview window handle associated with the capture device, if any. * * @param id The capture device ID. * * @return The window ID of the preview window for the * specified capture device ID, or NULL if preview * does not exist. */ PJ_DECL(pjsua_vid_win_id) pjsua_vid_preview_get_win(pjmedia_vid_dev_index id); /** * Stop video preview. * * @param id The capture device ID. * * @return PJ_SUCCESS on success, or the appropriate error code. */ PJ_DECL(pj_status_t) pjsua_vid_preview_stop(pjmedia_vid_dev_index id); }}} === Video (Account) Configuration === #acc Video settings are mostly configured on {{{pjsua_acc_config}}} with the following fields: {{{ /** * Maximum number of simultaneous active video streams to be allowed * for calls on this account. Setting this to zero will disable video * in calls on this account, regardless of other video settings. * * Default: 1 */ unsigned max_video_cnt; /** * Specify whether incoming video should be shown to screen by default. * This applies to incoming call (INVITE), incoming re-INVITE, and * incoming UPDATE requests. * * Regardless of this setting, application can detect incoming video * by implementing \a on_call_media_state() callback and enumerating * the media stream(s) with #pjsua_call_get_info(). Once incoming * video is recognised, application may retrieve the window associated * with the incoming video and show or hide it with * #pjsua_vid_win_set_show(). * * Default: PJ_FALSE */ pj_bool_t vid_in_auto_show; /** * Specify whether outgoing video should be activated by default when * making outgoing calls and/or when incoming video is detected. This * applies to incoming and outgoing calls, incoming re-INVITE, and * incoming UPDATE. If the setting is non-zero, outgoing video * transmission will be started as soon as response to these requests * is sent (or received). * * Regardless of the value of this setting, application can start and * stop outgoing video transmission with #pjsua_call_set_vid_strm(). * * Default: PJ_FALSE */ pj_bool_t vid_out_auto_transmit; /** * Specify the default capture device to be used by this account. If * \a vid_out_auto_transmit is enabled, this device will be used for * capturing video. * * Default: PJMEDIA_VID_DEFAULT_CAPTURE_DEV */ pjmedia_vid_dev_index vid_cap_dev; /** * Specify the default rendering device to be used by this account. * * Default: PJMEDIA_VID_DEFAULT_RENDER_DEV */ pjmedia_vid_dev_index vid_rend_dev; }}} === Video Call Manipulation === #vcm The default video behavior for a call is controlled by the account settings above. On top of that, the application can manipulate video of an already-going call by using {{{pjsua_call_set_vid_strm()}}} API: {{{ /** * This enumeration represents video stream operation on a call. * See also #pjsua_call_vid_strm_op_param for further info. */ typedef enum pjsua_call_vid_strm_op { /** * Add a new video stream. This will add a new m=video line to * the media, regardless of whether existing video is/are present * or not. This will cause re-INVITE or UPDATE to be sent to remote * party. The number of maximum active video streams in a call is * still limited by \a max_video_cnt setting in pjsua_acc_config. */ PJSUA_CALL_VID_STRM_ADD, /** * Remove/disable an existing video stream. This will * cause re-INVITE or UPDATE to be sent to remote party. */ PJSUA_CALL_VID_STRM_REMOVE, /** * Change direction of a video stream. This operation can be used * to activate or deactivate an existing video media. This will * cause re-INVITE or UPDATE to be sent to remote party. */ PJSUA_CALL_VID_STRM_CHANGE_DIR, /** * Change capture device of a video stream. This will not send * re-INVITE or UPDATE to remote party. */ PJSUA_CALL_VID_STRM_CHANGE_CAP_DEV, /** * Start transmitting video stream. This will cause previously * stopped stream to start transmitting again. Note that no * re-INVITE/UPDATE is to be transmitted to remote since this * operation only operates on local stream. */ PJSUA_CALL_VID_STRM_START_TRANSMIT, /** * Stop transmitting video stream. This will cause the stream to * be paused in TX direction, causing it to stop sending any video * packets. No re-INVITE/UPDATE is to be transmitted to remote * with this operation. */ PJSUA_CALL_VID_STRM_STOP_TRANSMIT, } pjsua_call_vid_strm_op; /** * Parameters for video stream operation on a call. */ typedef struct pjsua_call_vid_strm_op_param { /** * Specify the media stream index. This can be set to -1 to denote * the default video stream in the call, which is the first active * video stream or any first video stream if none is active. * * This field is valid for all video stream operations, except * PJSUA_CALL_VID_STRM_ADD. * * Default: -1 (first active video stream, or any first video stream * if none is active) */ int med_idx; /** * Specify the media stream direction. * * This field is valid for the following video stream operations: * PJSUA_CALL_VID_STRM_ADD and PJSUA_CALL_VID_STRM_CHANGE_DIR. * * Default: PJMEDIA_DIR_ENCODING_DECODING */ pjmedia_dir dir; /** * Specify the video capture device ID. This can be set to * PJMEDIA_VID_DEFAULT_CAPTURE_DEV to specify the default capture * device as configured in the account. * * This field is valid for the following video stream operations: * PJSUA_CALL_VID_STRM_ADD and PJSUA_CALL_VID_STRM_CHANGE_CAP_DEV. * * Default: capture device configured in account. */ pjmedia_vid_dev_index cap_dev; } pjsua_call_vid_strm_op_param; /** * Add, remove, modify, and/or manipulate video media stream for the * specified call. This may trigger a re-INVITE or UPDATE to be sent * for the call. * * @param call_id Call identification. * @param op The video stream operation to be performed, * possible values are #pjsua_call_vid_strm_op. * @param param The parameters for the video stream operation, * or NULL for the default parameter values * (see #pjsua_call_vid_strm_op_param). * * @return PJ_SUCCESS on success or the appropriate error. */ PJ_DECL(pj_status_t) pjsua_call_set_vid_strm ( pjsua_call_id call_id, pjsua_call_vid_strm_op op, const pjsua_call_vid_strm_op_param *param); /** * Get the media stream index of the default video stream in the call. * Typically this will just retrieve the stream index of the first * activated video stream in the call. * * @param call_id Call identification. * * @return The media stream index or -1 if no video stream * is present in the call. */ PJ_DECL(int) pjsua_call_get_vid_stream_idx(pjsua_call_id call_id); }}} === Video Call Information === #vci New fields have been added to {{{pjsua_call_info}}} to report media information in more detail: Snippet of {{{pjsua_call_info}}} new fields: {{{ /** Number of media streams in this call */ unsigned media_cnt; /** Array of media stream information */ struct { /** Media index in SDP. */ unsigned index; /** Media type. */ pjmedia_type type; /** Media direction. */ pjmedia_dir dir; /** Call media status. */ pjsua_call_media_status status; /** The specific media stream info. */ union { /** Audio stream */ struct { /** The conference port number for the call. */ pjsua_conf_port_id conf_slot; } aud; /** Video stream */ struct { /** * The window id for incoming video, if any, or * PJSUA_INVALID_ID. */ pjsua_vid_win_id win_in; /** The video capture device for outgoing transmission, * if any, or PJMEDIA_VID_INVALID_DEV */ pjmedia_vid_dev_index cap_dev; } vid; } stream; } media[PJMEDIA_MAX_SDP_MEDIA]; }}} === Video Call Stream Information and Statistic === #vcsi New API has been added to query call's stream information and statistic. Subsequently, the old {{{pjsua_call_get_media_session()}}} API has been deprecated since its use is unsafe. New API: {{{ /** * Media stream info. */ typedef struct pjsua_stream_info { /** Media type of this stream. */ pjmedia_type type; /** Stream info (union). */ union { /** Audio stream info */ pjmedia_stream_info aud; /** Video stream info */ pjmedia_vid_stream_info vid; } info; } pjsua_stream_info; /** * Media stream statistic. */ typedef struct pjsua_stream_stat { /** RTCP statistic. */ pjmedia_rtcp_stat rtcp; /** Jitter buffer statistic. */ pjmedia_jb_state jbuf; } pjsua_stream_stat; /** * Get media stream info for the specified media index. * * @param call_id The call identification. * @param med_idx Media stream index. * @param psi To be filled with the stream info. * * @return PJ_SUCCESS on success or the appropriate error. */ PJ_DECL(pj_status_t) pjsua_call_get_stream_info(pjsua_call_id call_id, unsigned med_idx, pjsua_stream_info *psi); /** * Get media stream statistic for the specified media index. * * @param call_id The call identification. * @param med_idx Media stream index. * @param psi To be filled with the stream statistic. * * @return PJ_SUCCESS on success or the appropriate error. */ PJ_DECL(pj_status_t) pjsua_call_get_stream_stat(pjsua_call_id call_id, unsigned med_idx, pjsua_stream_stat *stat); /** * Get media transport info for the specified media index. * * @param call_id The call identification. * @param med_idx Media stream index. * @param t To be filled with the transport info. * * @return PJ_SUCCESS on success or the appropriate error. */ PJ_DECL(pj_status_t) pjsua_call_get_med_transport_info(pjsua_call_id call_id, unsigned med_idx, pjmedia_transport_info *t); }}} === Video Window API === #vw A video window is a rectangular area in your monitor to display video content. The video content may come from remote stream, local camera (in case of preview), AVI playback, or any other video playback. Application mostly will be interested in the native handle of the video window so that it can embed it in its application window, however we also provide simple and commonly used API for manipulating the window. {{{ /** * This structure describes video window info. */ typedef struct pjsua_vid_win_info { /** * Renderer device ID. */ pjmedia_vid_dev_index rdr_dev; /** * Native window handle. */ pjmedia_vid_dev_hwnd hwnd; /** * Window show status. The window is hidden if false. */ pj_bool_t show; /** * Window position. */ pjmedia_coord pos; /** * Window size. */ pjmedia_rect_size size; } pjsua_vid_win_info; /** * Enumerates all video windows. * * @param id Array of window ID to be initialized. * @param count On input, specifies max elements in the array. * On return, it contains actual number of elements * that have been initialized. * * @return PJ_SUCCESS on success, or the appropriate error code. */ PJ_DECL(pj_status_t) pjsua_vid_enum_wins(pjsua_vid_win_id wids[], unsigned *count); /** * Get window info. * * @param wid The video window ID. * @param wi The video window info to be initialized. * * @return PJ_SUCCESS on success, or the appropriate error code. */ PJ_DECL(pj_status_t) pjsua_vid_win_get_info(pjsua_vid_win_id wid, pjsua_vid_win_info *wi); /** * Show or hide window. * * @param wid The video window ID. * @param show Set to PJ_TRUE to show the window, PJ_FALSE to * hide the window. * * @return PJ_SUCCESS on success, or the appropriate error code. */ PJ_DECL(pj_status_t) pjsua_vid_win_set_show(pjsua_vid_win_id wid, pj_bool_t show); /** * Set video window position. * * @param wid The video window ID. * @param pos The window position. * * @return PJ_SUCCESS on success, or the appropriate error code. */ PJ_DECL(pj_status_t) pjsua_vid_win_set_pos(pjsua_vid_win_id wid, const pjmedia_coord *pos); /** * Resize window. * * @param wid The video window ID. * @param size The new window size. * * @return PJ_SUCCESS on success, or the appropriate error code. */ PJ_DECL(pj_status_t) pjsua_vid_win_set_size(pjsua_vid_win_id wid, const pjmedia_rect_size *size); }}} === Video Codec API === #vc API for managing video codecs: {{{ /** * Enum all supported video codecs in the system. * * @param id Array of ID to be initialized. * @param count On input, specifies max elements in the array. * On return, it contains actual number of elements * that have been initialized. * * @return PJ_SUCCESS on success, or the appropriate error code. */ PJ_DECL(pj_status_t) pjsua_vid_enum_codecs( pjsua_codec_info id[], unsigned *count ); /** * Change video codec priority. * * @param codec_id Codec ID, which is a string that uniquely identify * the codec (such as "H263/90000"). Please see pjsua * manual or pjmedia codec reference for details. * @param priority Codec priority, 0-255, where zero means to disable * the codec. * * @return PJ_SUCCESS on success, or the appropriate error code. */ PJ_DECL(pj_status_t) pjsua_vid_codec_set_priority( const pj_str_t *codec_id, pj_uint8_t priority ); /** * Get video codec parameters. * * @param codec_id Codec ID. * @param param Structure to receive video codec parameters. * * @return PJ_SUCCESS on success, or the appropriate error code. */ PJ_DECL(pj_status_t) pjsua_vid_codec_get_param( const pj_str_t *codec_id, pjmedia_vid_codec_param *param); /** * Set video codec parameters. * * @param codec_id Codec ID. * @param param Codec parameter to set. Set to NULL to reset * codec parameter to library default settings. * * @return PJ_SUCCESS on success, or the appropriate error code. */ PJ_DECL(pj_status_t) pjsua_vid_codec_set_param( const pj_str_t *codec_id, const pjmedia_vid_codec_param *param); }}} {{{ #!html
}}}