Custom Query (2195 matches)

The pjsua will always restart the media transport whenever media description gets updated, e.g: SDP renegotiation in re-INVITE or UPDATE, hence the SRTP state will also get restarted, which is not desired, e.g: the ROC (Roll Over Counter) of SRTP state may get out of sync.

Scenario:

• we receive INVITE
• we send 200/OK with G.722.1 (pt=119) as the selected codec
• call is confirmed, media is OK
• we disabled all codecs except G.722 (pt=9)
• we send UPDATE
• we receive 401, we resend UPDATE
• we receive 200/OK for UPDATE, with G.722 codec (pt=9) in the answer
• at this point, we should be using G.722 codec (pt=9) in the media, but it looks like we're still expecting G.722.1:

  09:29:01.429  strm0x12ebca8  Bad RTP pt 9 (expecting 119)
  

Here is the scenario to reproduce:

1. Receiving offer with one media line disabled and no rtpmap for dynamic PT (valid):

   m=audio 5000 RTP/AVP 0
   m=video 0 RTP/AVP 100
   

2. PJSUA answers similarly (no rtpmap for dynamic PT), e.g:

   m=audio 4000 RTP/AVP 0
   a=rtcp:4001 IN IP4 192.168.1.30
   a=rtpmap:0 PCMU/8000
   a=sendrecv
   m=video 0 RTP/AVP 100
   

3. PJSUA sends re-INVITE (using latest active local SDP as above), after receiving the answer, assertion raised in SDP negotiation.

The SIPp scenario can be found in tests/pjsua/scripts-sipp/uac-inv-two-media-but-one-disabled-no-rtpmap.xml.

Ticket #883 added a new keep-alive mechanism for the stream, which would still be performed even when the stream is being paused. Unfortunately, no packet is transmitted on the RTCP port in this situation.

This ticket would enable packet transmission for this situation. With this ticket, an RTCP packet would be transmitted periodically as long as PJMEDIA_STREAM_ENABLE_KA is set to non-zero (note that by default it is disabled).

to save:

• the jitter buffer
• RTCP stat

Ticket #883 added another keep-alive mechanism to media stream, which has to be selected at compile time.

This ticket proposes to add these configuration, and probably other keep-alive mechanisms, into the new run-time configurable settings.

The settings would most likely be part of pjmedia_stream_info, and on PJSUA-LIB it will be part of account settings.

Updates suggested:

1. Currently the RFC 3550 compliant jitter calculation is used in generating RTCP RR packet, however the jitter statistics is calculated from "raw jitter" (referring to "different D" of RFC 3550), so changes to be done:
   • jitter statistics should use RFC 3550
   • add "raw jitter" statistics (calculated from absoluted RFC 3550 diff D), compile-time configurable, disabled by default.
   • also add IPDV (calculated from RFC 3550 diff D), compile-time configurable, disabled by default.
2. Add feature to reset RTCP statistics.

It was very noticeable on music streaming, barely noticeable on speech.

The pjmedia G722.1 implementation applies signal adjustment before encoder and after decoder (configurable via PJMEDIA_G7221_DEFAULT_PCM_SHIFT), while IPP's implementation doesn't.

When local VAD is enabled but then the remote signals "annexb=no" (no VAD/DTX support) in the SDP, IPP G.729 still generates and sends silence or DTX packets.

At first glance, the codec handles "annexb=no" correctly, i.e: the VAD is turned off, however it doesn't update its VAD status to the application, so the application is still assuming that the VAD is active.

The problem was examined after the stream module performing NAT hole punching in the beginning of call by disabling and soon re-enabling the VAD.

Detail report (and fix) can be found ​here.

There were a couple of reports saying that echo cancellation works fine using SDK 3.x but doesn't work properly when using SDK 4.x. (on devices running iOS 3.x and 4.x).

This ticket investigates whether there is a problem with EC using SDK 4.x, and if possible, finds a way to solve it.

Note: using speaker mode is recommended to test the EC.

Currently, the bytes_per_frame in stream port info contains encoded frame size info, regardless if data flow from/to stream (via get_frame()/put_frame()) is encoded or raw data.

This may cause problem as any pjmedia ports may use bytes_per_frame info, e.g: resample port uses bytes_per_frame info of down stream port to calculate dynamic buffer size, and lesser buffer size may cause buffer overrun.

The original report can be found ​here.

Call to AudioConverterNew() for the iLBC codec may take long time to complete, causing timeout in acquiring pjsua lock (when doing get call info on incoming call).

This happens on iPhone 3GS with iOS 4.0.2 and SDK 4.0.

Ticket #883 added compile time configuration for selecting keep-alive mechanism of the media streams. Some applications want more flexibility by configuring this at run-time, ideally by making this part of account settings.

This ticket will add run-time configuration for activating/deactivating stream keep-alive only (no keep-alive packet type and interval settings).

Activating/deactivating stream keep-alive mechanism in run-time

• Make sure that stream keep-alive mechanism is enabled:
  • define PJMEDIA_STREAM_ENABLE_KA to PJMEDIA_STREAM_KA_EMPTY_RTP or PJMEDIA_STREAM_KA_USER in config_site.h
  • optionally, update default keep-alive settings (user packet content and interval) via compile-time macros PJMEDIA_STREAM_KA_USER_PKT and PJMEDIA_STREAM_KA_INTERVAL.
• If you are using PJSUA, configure this via account config:
  • set pjsua_acc_config.use_stream_ka
  • use the account config for adding (pjsua_acc_add()) or modifying (pjsua_acc_modify()) account, so all media streams for that account will use this setting.
• If you are using PJMEDIA level, set stream info pjmedia_stream_info.use_ka and use the stream info for creating stream (pjmedia_stream_create()).

The proposed patch can be found ​here

Currently padding bit in incoming RTP packets are ignored, potentially causing incorrect RTP payload processing.

Also investigate if the padding bit should be used in outgoing RTP transmission.

Reported that this occurred on "slow CPU" (iPhone 3G) in "heavy CPU load" and barely reproducible.

The stack when crashing:

0   0x0018bdb4 auth_get_tag_length (auth.c:64)
1   0x001898dc srtp_unprotect (srtp.c:885)
2   0x00135310 srtp_rtp_cb (transport_srtp.c:864)
3   0x0011ce70 ice_on_rx_data (transport_ice.c:1641)
4   0x0014907c stun_on_rx_data (ice_strans.c:1395)
5   0x001404ac on_data_recvfrom (stun_sock.c:815)
6   0x000f08a0 ioqueue_on_read_complete (activesock.c:573)
7   0x000fb4bc ioqueue_dispatch_read_event (ioqueue_common_abs.c:560)
8   0x000fbb7c pj_ioqueue_poll

After investigation, a possible scenario is:

1. the call is disconnecting, so media deinitialization steps are on progress
2. an RTP packet is coming and ice_on_rx_data() is about to call SRTP callback rtp_cb in media worker thread
3. context switch happens, now pjsua_media_channel_deinit() is executed (at least until pjmedia_transport_media_stop()) in signaling worker thread
4. when the context back to media worker thread, SRTP callback is called after SRTP backend states (or even the SRTP instance itself) have been destroyed

When SDP media in the offer uses port zero, it means that the media is not wanted and must be answered with port zero as well (RFC 3264 Section 5.1 and 8.2). However, some implementations may disobey it (we found one!), and unfortunately it may lead to assertion in the SDP negotiation process (when there is dynamic payload type involved).

Sample SDP offer/answer that trigger assertion:

• Offer (may be after some offer/answer cycles):

  ...
  m=audio 4000 RTP/AVP 0
  m=video 0 RTP/AVP 100
  ...
  

• Answer:

  ...
  m=audio 6000 RTP/AVP 0
  m=video 7000 RTP/AVP 100
  a=rtpmap:100 H261/90000
  ...
  

Compile errors are something like:

pjmedia-codec\types.h(60): 'PJMEDIA_RTP_PT_DYNAMIC' : undeclared identifier
pjmedia-codec\types.h(60): expected constant expression

Note: telephone events can be disabled by setting PJMEDIA_RTP_PT_TELEPHONE_EVENTS to zero.

It has been reported that some operators in Japan have trouble processing this feature, hence a run-time setting to enable/disable this is needed.

We validate incoming RTP packets to see if they contain correct payload type, as it should. However some people are having problems with broken servers that requires them to disable this checking.

The original report can be found ​here.

Issues reported:

• codec description pool in codec manager is released before the new setting is copied, this causes problem when the new setting contains data (e.g: fmtp strings) that is allocated using that pool,
• dec_fmtp is copied twice, one of them should be enc_fmtp instead.

Crash in coreaudio_dev when VoIP call is hung up during an interruption event. This is related to ticket #1172.

The fix also adds support if there are more than one active streams that are interrupted.

As mentioned in RFC3711 5.2 & RFC4568 6.2.

There is MIME subtype AAL2-G726-xx for the packetization variants

• Either extend existing stream.c to support video, or create a new vstream.c.
• Use vstreamutil.c as demo.

There is already video stream implementation in the stream.c and sample in the vstreamutil.c, however it's just proof-of-concept version, so the to do/consider list will be:

• proper picture bitstream generation from RTP packet, currently the video stream will just 'concatenate' incoming RTP to get a complete picture bitstream, and everytime it got end-of-picture flag, RTP header m==1, push the picture into jitter buffer, so in case of unordered RTP the picture bitstreams will be truncated
• perhaps jitter buffer frame content/granularity should be RTP payload instead of a picture bitstream (picture bitstream will be generated right before decoding, instead of inline with incoming RTP packets), however current jitter buffer only has RTP sequence (no RTP timestamp), so need to research if the current jitter buffer fits this requirements along with optimization consideration (e.g: frame size, minimal memcpy)
• audio-video sync
• encoder & decoder size can be different, e.g: remote endpoint, which is an embedded/mobile, only supports QCIF while local endpoint, which is desktop, prefers CIF, so the issues will be:
  • which size should be put in the stream info (as currently it only has one 'format info' field)?
  • the decoder output format (e.g: format id & size) can only be ensured after a successful picture decoding (the format setting in opening the decoder is rather unusable and the local SDP may offer multiple sizes), need a way to notify application about the active format.

• generation of SDP
  • how codec can put codec specific info in SDP?
• codec specific SDP negotiation, if any
• applying parameters in SDP to the codec

TODO:

1. add video SDP generation in pjmedia_endpt_create_sdp()
2. add video stream info generation from SDP in pjmedia_stream_info_from_sdp()
3. add codec specific negotiation, perhaps sdp_neg.c should have external format negotiation mechanism to handle format specific negotiation, e.g: H26x picture resolution negotiation, this negotiation result will be applied to codec param (for stream in opening the codec)
4. add auto-assignment mechanism for dynamic payload type (in endpoint when creating SDP?), currently, the space for hardcoded dynamic payload type is very tight (or even not enough for video formats)
5. auto apply (or at least validate) SDP fmtp setting to codec param (or the other way around?), e.g: when SDP fmtp setting in codec is set to "CIF=1" codec param should be set to "size=352x288 & fps=30/1.001"
6. add video capability to simpleua to test video stuffs on real SIP application

Audio flood/burst in the beginning of call triggers jitter buffer progressive discard (see ticket #969) to start dropping frames intensively. In the decoding/playing, the dropped frames introduce unaligned signal, so the audio breakups occur.

The proposed solution is by replacing the frames next to the dropped/discarded frames with 'empty' frame to trigger PLC invocation in the decoding so the audio signal will be smoother/aligned by the PLC.

The jitter buffer has an algorithm to discard few frames to improve latency when its contents are exceeding certain limit. This algorithm is called progressive discard and is enabled by default.

Unfortunately, this discard operation causes click noise to be produced, since the PCM wave signal of the next frame is no longer aligned with the previous frame.

This patch proposes dropping the next frame after the discarded frame, to cause invocation of the PLC, which would produce smoother wave signal than the original algorithm, thereby removing the click noise.

On iOS, application can now set the audio I/O latency by using PJMEDIA_AUD_DEV_CAP_INPUT_LATENCY / PJMEDIA_AUD_DEV_CAP_OUTPUT_LATENCY, which can set the latency (via Audio Session's property kAudioSessionProperty_PreferredHardwareIOBufferDuration) to as low as 5 ms.

coreaudio_dev now supports recording using a different sample rate than the hardware's sample rate by using Audio Converter for sample rate conversion.

When compiling using iOS SDK 4.x, the echo cancellation and support for Bluetooth input do not work properly. This is related to ticket #1113.

On Mac OS X 10.6, when audio input/output callback thread is changed (such as when plugging/unplugging headphone), the input/output callback will trigger an assertion when calling pj_thread_register().

Currently, coreaudio backend will use the first input/output audio device as the default. It is more preferable if we can use the system's default input/output audio device instead.

There is G722 implementation that uses 10ms packet length. As currently the pjmedia G722 wrapper expects G722 packet length to be (multiple of) 20ms, the incoming 10ms G722 packets cannot be parsed properly, which leads to:

• no audio
• divide-by-zero exception

This feature adds support for pjmedia to refresh its audio device list, thereby allowing application to use/stop using audio devices that are plugged/unplugged when the application is running.

Application would still need to do its own implementation to detect audio devices plug/unplug event. Then it can call the function pjmedia_aud_dev_refresh().

Note that this feature is not implemented on PortAudio, legacy, and all Symbian sound backends.

Add an API for pjmedia_stream to get the last frame type retrieved from the jitter buffer.

The use of pjmedia_session in <pjmedia/session.h> is now deprecated. Application should directly create the audio and video streams instead.

The pjmedia_session will continue to exist but this will only support audio session.

Applications that use pjmedia_session would still expected to work, but you should change it as soon as possible.

• Setting audio session category is now during stream creation instead of in the factory initialization.
• Reset the audio session category after an interruption.
• By default, audio route change property listener is disabled as it is no longer required.

This is placeholder ticket for all works related to video devices.

Work on the framework..

Backport of ticket #1200 to the 2.0-dev branch.

There are some implementations that put no format list on disabled/rejected media, e.g:

m=audio 0 RTP/AVP
m=audio 2000 RTP/AVP 0

The SDP parser will return RTP/AVP\nm=audio for the transport of the first SDP media.

Missing keyframe is a major and common issue in real-time video session. Packet lost or discarding packets by jitter buffer (opening video devs may take few seconds) in the beginning of video session could be the main cause.

We experienced this problem a lot in sipit! When using H.264, missing keyframe causes pjsua got no video (decoding failure), sometime video could be successfully decoded after call hold and resume. When using H.263, initial video came foggy (only the moving object part had clear view, background/other part was gray).

Solutions/ideas:

1. Prevention:
   • for receiving direction:
     • prevent jitter buffer to discard first packet and following packets with same timestamp,
     • and/or, as long as there is space in jitter buffer, avoid dropping any video packets, just adjust the playback time (make it faster/slower), if possible.
   • for sending direction: force sending keyframes in some short periods in the beginning of video session for few seconds.
2. Implementing media feedback/control mechanisms (for both directions):
   • Full Intra Request in ​RFC 5104 3.5.1
   • Picture Loss Indication feedback in ​RFC 4585 6.3.1
   • Fast Update Command for the Full Picture in ​RFC 5168 7.1. Note that some implementations refuse to send us SIP INFO request because we don't list "INFO" in our SIP header "Allow".

Note:

• Missing keyframe detection has been implemented in this ticket along with SIP INFO mechanism.
• Check #1437 for RTCP-FB PLI mechanism.

Known cases:

1. offering multiple H.264, with PT=xx and yy, answered with single H.264 with PT=zz, our H.264 SDP negotiator deduce that "zz" match to "xx", but incoming RTP packets use "yy" instead! Note that this also happened before r3526 (permissive H.264 negotiation).
2. receiving offer with PT=xx, answering with PT=yy, then peer refuse our RTP packets with PT=xx, peer expects our RTP packets using PT=yy instead! (fixed by #1300)

Scenario:

1. In SRTP mandatory mode, call established with 2 media: active audio with SRTP enabled and inactive video (port 0).
2. Remote send re-offer with three SDP media lines: audio with RTP/SAVP+crypto, deactivated video (port 0) with RTP/SAVP without crypto, audio with RTP/AVP without crypto.
3. We reject the re-INVITE with 488 (actual error code PJMEDIA_SRTP_ESDPREQCRYPTO).

It should not be rejected as the first media in the reoffer is actually acceptable.

Polycom DMA rejects our video offer because it's missing "max-fs" H264 parameter. They need it to calculate bandwidth needed. See also #1244

According to ​RFC 5245: If RTCP is not in use, the agent MUST signal that using b=RS:0 and b=RR:0 attributes as defined in ​RFC 3556.

API to set bandwidth.

Also related to #1239, bandwidth attribute is needed by some servers in order to let video calls through.

Ref: Avaya

When splitcomb's put_frame() is called with frame type NONE, it will generate zero-frame for its reversed channels, if any. The bug was in this zero-frame generation, i.e: bzero'd one-channel-size buffer with multichannel-size length.

Original report can be found ​here.

The API specification says that application may call pjmedia_aud_subsys_shutdown() more than once, as long as the number matches the number of times pjmedia_aud_subsys_init() is called. However doing so will lead to a crash in the second call to the function.

By default, echo canceller's delay buffer will make use of WSOLA to expand and shrink audio samples. By setting the option flag in pjsua_set_ec() to contain PJMEDIA_ECHO_USE_SIMPLE_FIFO, application can use simple FIFO mechanism instead.

There is bug in jb prefetching mechanism, the prefetch condition may cause unexpected JB state switches that will disturb burst level calculation, i.e: when JB empty condition occurs in initializing state, the state will be immediately switched to prefetching, and after the prefetch processing done, the state will be directly set to processing (instead of initializing). The expected behaviour is that JB state switch from initializing to processing is only done by jb_update().

This causes the following division by zero exception:

Program received signal SIGFPE, Arithmetic exception.
[Switching to Thread 0xb73a5b90 (LWP 7148)]
0x08184372 in pjmedia_jbuf_put_frame2 (jb=0x8323cf8, frame=0x826f86c, frame_size=80, bit_info=0, frame_seq=21, discarded=0xb73a502c)
   at ../src/pjmedia/jbuf.c:763
763                 burst_factor = cur_size / burst_level;
(gdb)

PJSUA-LIB video API specification:

1. Video Device Enumeration API

Specifications:

1. query device count
2. query device names
3. hotswap support?

2. Video Call API

Specifications:

1. enable or disable video when initiating call
2. enable or disable video when receiving call
3. choose which device to use for video call
4. add and remove video media stream during the call
5. change which device to use during the call
6. hide remote video window
7. move and/or resize video window

Notes:

Q: when receiving incoming call or reinvite which adds video, how can the app confirm this to the user?
A: at the moment, incoming video will be accepted automatically, but the app may simply hide the window until it gets user confirmation, if needed.

Q: when receiving incoming call or reinvite with video, should we start sending automatically?
A: at the moment the answer tends to be no, as sending video is an operation that can be initiated independently.

3. Video Capture Preview API

Specifications:

1. start and stop preview for the specified camera device, independent of calls
2. preview more than one capture devices independently
3. move and/or resize preview window

4. Media Info and Statistic API

The new API can be used for both audio and video.

Specifications:

1. query the number of media streams in a call
2. query the media stream info
3. query the media stream statistic
4. query the media transport info/statistic
5. move pjsua function that dumps call statistic (i.e. "dq" command) to a separate size, to reduce footprint for apps that don't use it.

5. Notification API

Nothing special for video. Update in media status is reported in on_media_state() callback as usual. Application may use media status API to query about video status.

By default, RTCP SDES/BYE is automatically sent by the stream. By setting the field pjmedia_stream_info.rtcp_sdes_bye_disabled to PJ_TRUE, application can choose to manually send RTCP SDES/BYE by using the API pjmedia_stream_send_rtcp_sdes() and pjmedia_stream_send_rtcp_bye().

Hardcoded format list in the above components, which may not reflect the actual capability of the component.

Rules about who should generate the timestamp is not clear. There may also be places where timestamp information is regenerated.

New API to register all known codecs that are implemented in PJMEDIA-CODEC, and to automatically deregister them on shutdown:

/* file <pjmedia-codec/audio_codecs.h> */

/**
 * Codec configuration. Call #pjmedia_audio_codec_config_default() to initialize
 * this structure with the default values.
 */
typedef struct pjmedia_audio_codec_config
{
    /** Speex codec settings. See #pjmedia_codec_speex_init() for more info */
    struct {
	unsigned	option;		/**< Bitmask of options.	*/
	unsigned	quality;	/**< Codec quality.		*/
	unsigned	complexity;	/**< Codec complexity.		*/
    } speex;

    /** iLBC settings */
    struct {
	unsigned	mode;		/**< iLBC mode.			*/
    } ilbc;

    /** Passthrough */
    struct {
	pjmedia_codec_passthrough_setting setting; /**< Passthrough	*/
    } passthrough;

} pjmedia_audio_codec_config;


/**
 * Initialize pjmedia_audio_codec_config structure with default values.
 *
 * @param cfg		The codec config to be initialized.
 */
PJ_DECL(void) pjmedia_audio_codec_config_default(pjmedia_audio_codec_config *cfg);

/**
 * Register all known audio codecs implemented in PJMEDA-CODEC library to the
 * specified media endpoint.
 *
 * @param endpt		The media endpoint.
 * @param c		Optional codec configuration, or NULL to use default
 * 			values.
 *
 * @return		PJ_SUCCESS on success or the appropriate error code.
 */
PJ_DECL(pj_status_t)
pjmedia_codec_register_audio_codecs(pjmedia_endpt *endpt,
                                    const pjmedia_audio_codec_config *c);

Note to codec writers:

A new mandatory destroy callback has been added to factory operation. You must implement this callback (hint: just point it to your existing function that deinitialize your codec factory).

Note to app writers:

You don't need to initialize each codec now, just use pjmedia_codec_register_audio_codecs() above. Similarly you should remove all the calls to codec deinitialization routines since that will be done automatically when the media endpoint is destroyed (leaving these calls as they are is harmless though; they will render to no-ops).

A thin wrapper for codec operations to make the API slightly cleaner.

It's probably more useful for audio codecs since we're running out of dynamic payload type constants. Also, allowing application to customize codec PT allocation will be nice.

Answerer SHOULD use matching payload type as the offer:

• RFC 3264 6.1:

In the case of RTP, if a particular codec was referenced with a specific payload type number in the offer, that same payload type number SHOULD be used for that codec in the answer.

• RFC 6184:

To simplify the handling and matching of these configurations, the same RTP payload type number used in the offer SHOULD also be used in the answer, as specified in [8]. An answer MUST NOT contain the payload type number used in the offer unless the configuration is exactly the same as in the offer.

Note:

• Implemented in SDP negotiator (instead of in endpoint as the preanswer generator), as the codecs need to be negotiated first.
• Initiating re-INVITE will revert the payload type numbers to the pjmedia defaults, which should be fine. If this is not acceptable, a simple solution will be providing alternatives for SDP generation for the re-INVITE:
  • using active local SDP,
  • no SDP.

When the PCM signal is in the rage -7 to -1 inclusive, wrong alaw value will be generated by the non-table version of pjmedia_linear2alaw() function.

Note that the table version of the function, which is the default implementation used by pjmedia, is not affected by this bug.

Symptoms:

• Call with AMR between IPP backend and S60 APS/VAS-direct with VAD turned on, loud noise heard on IPP side when S60 side stops talking.
• Call with AMR among IPP backends with native IPP AMR VAD turned on (by default, it's using PJMEDIA VAD), loud noise on one side whenever the remote side is silence.

After investigation, we found some bugs related to AMR SID/NO_DATA frame packetization:

• SID type indicator (STI) bit is not set/get properly in AMR SID frame.
• Bad AMR data length definition for AMR NO_DATA frame.

Found that setting the output volume when the VAS engine hasn't been completely started will be useless (no output volume change).

PJMEDIA object will be given object signature. This is needed for the event framework so that application can at least distinguish the sender of the event.

Develop a really simple GUI app as proof of concept that the video API works

Review the current videodev API to manage window (resize, move, hide, etc.) against window handle passing approach.

Use the simple GUI app (#1327) for testing.

Currently, preview API only exists in PJSUA-LIB, and it is done by creating a rendering device, videoport, and vide tee to move the frames from the capture to render device.

As it turns out, the capture device on iOS has the capability to create preview layer natively, hence there is no need to create a rendering device and feed the frames manually. While the current PJSUA-LIB API is still able to accommodate this, it may be cleaner to implement this on pjmedia-videodev layer.

The alternative solution for current API is something like this:

• add new capability in video capture device to specify whether it has native preview layer or not
• create preview API in pjmedia-videodev. The preview API will work with device's native preview layer if it is available, or create a rendering device when it's not.

This is caused by a bug in samples-per-frame calculation PJMEDIA_SPF(). So applications that use PJMEDIA_SPF() for allocating buffer will get "too small buffer size" error or assertion from codec.

When a new render window (either preview or stream) is created and currently another render window is active, sometime both rendering processes got stuck. After manually resizing main window, all rendering processes continue normally.

The problem only occurred on Windows, Mac is unaffected.

Crash on receiving UPDATE or re-INVITE.

Background

• Mobile devices can be rotated (the rotation angle can be detected), which may lead to wrong video orientation in rendering/playback.

Objective

• The purpose is to allow video rendering in natural orientation (head/sky upside) or the preferred orientation of the capturer.

Notes

• Common video codecs works in predefined allowed sizes/orientation, e.g: CIF, QCIF.
• On iOS, capturing video to a file will store the video data in camera's native orientation, i.e: wide-oriented, the capture orientation info (the device/camera orientation when the capture/record process is done) is stored in video metadata, so in playing back the file, the stored capture orientation info (may be combined with current device orientation info) is used to calculate rotation angle to get natural orientation in video rendering/playback.

Ideas

• The orientation info is readable and read-only in capture.
• The renderer will apply the orientation info provided by the capturer.
• The orientation info is a rotation angle to be applied to a picture/video so it can be displayed naturally (or as the capturer preference).

Issues

• SDP/SIP only offers limited support for orientation, and there is no way for sender to tell receiver about the video orientation of the capture device. The 'a=orient:x' in SDP is not really useful as this is receiver preference/capability, while in practice, the receiver can render any orientations and the capturer cannot really send portrait (codecs limitation, etc).

Specification

Based on the background above, the following API will be implemented by this ticket:

• Add orientation capability to video device with the following specification:
  • it is read-only for capture devices, and
  • it is write-only for renderer devices
• Add pjsua_vid_win_rotate(win, angle) API. The angle is a rotation angle to be applied to a picture/video so it can be displayed naturally (or as the capturer preference).

Currently the UDP and ICE media transport has the capability to switch destination RTP/RTCP address to the source address of the RTP/RTCP packets, if they are different than the original address set in SDP received from remote.

The problem is audio playback is broken if more than one remotes are transmitting to us simultaneously, e.g. on early media forking scenario.

This ticket modifies the behavior so that the transmission won't switch as long as we're receiving RTP/RTCP packets from the "correct" remote address. Also when the remote peer address is still in probation, RTP/RTCP packets from that address will be silently discarded, to avoid playback confusion.

Note that there is a more ambitious proposal in #865.

Issues with current algorithm:

• Precision issue in discard rate calculation in JB progressive discard. When jb-size/delay equal to burst-level, jb frame discard should not occur (delay in JB is expected to be always equal or less than burst-level). But with current calculation, the JB will still discard some frames because of bad precision in discard rate calculation.
• The discard rates were predefined and hardcoded based on excessive latency, this may not suit well for some environments. So, there is an idea to calculate discard rate dynamically based on parameters such as actual burst level, excessive latency, aggressiveness setting.

Video codec framework employs SDP fmtp attribute for specifying codec specific parameters, e.g: MPI in H263, profile-level in H264. When SDP fmtp is not supplied, the codec will apply default values and this may override the 'standard' video parameters, such as size and frame rate, specified in pjmedia_format.

While application may want to specify the 'standard' video parameters only (no SDP fmtp) and don't want them overridden by default values because of empty/unsupplied SDP fmtp.

Sample cases:

• 'AMR/8000' (master/local-preanswer) vs 'AMR/8000/1' (slave/remote-offer): not match, while it should.
• 'AMR/8000/1' (master/local-preanswer) vs 'AMR/8000/2' (slave/remote-offer): match, while it shouldn't.

The original report can be found ​here.

The updated OpenCORE AMR instructions are in Using OpenCORE AMR NB and WB Codecs

Previously, we have problem to configure libx264, the ffmpeg h264 encoder backend, for realtime streaming, some libx264 settings are not really configurable from ffmpeg, e.g: h264 profile, NAL unit max size.

With the latest ffmpeg (from git/development, not released yet), app can configure all libx264 settings almost natively (same option names and raw libx264 option).

Image decoding is now done in on_rx_rtp() callback by the network thread as soon as full image is available in the jitter buffer.

The default behavior is to only decode one image and suspend further decoding until this image is consumed by get_frame(). The decoding can be made more aggressive by setting PJMEDIA_VID_STREAM_SKIP_PACKETS_TO_REDUCE_LATENCY to non-zero, and in this case the stream will always decode any available images in the jitter buffer regardless whether it will overwrite previously decoded image or not. This will notably improve latency, but if the network is too bursty several images may be skipped causing reduced fps playback and non-smooth video playback.

There is a synchronization/concurrency problem when stopping clock thread, namely the stop() function doesn't wait until the process in the clock callback completes. This causes problem for the user of the clock if it assumes that processing will complete when the clock is stopped.

Example:

m=video 55894 RTP/AVP 31 123
a=rtpmap:123 H264/90000
a=fmtp:123 profile-level-id=42801f;packetization-mode=0;level-asymmetry-allowed=1

Assertion line:

amr_helper.h:922: pjmedia_codec_amr_pack: Assertion `FT <= SID_FT || FT == 14 || FT == 15' failed.

After investigation, a fatal bug is found in ipp_codecs.c pack_amr() in setting up AMR-WB frame size, i.e: using AMR-NB frame size table instead of AMR-WB's.

This will cause wrong AMR-WB data pointers returned by the encoder (only the first frame gets the correct pointer) and the RTP payload generated (by AMR packetizer) will contain corrupted bitstream. So even if assertion is not raised, e.g: release mode or accidentally packetizer always read valid frame types in the wrong data pointer, this will definitely cause noise.

Note that in single frame per packet this issue is never raised as the first frame pointer always points to correct AMR bistream and the AMR packetizer doesn't really read the frame size info supplied by encoder (it uses AMR frame size lookup table instead).

Add support for creating an event manager that will accept subscription/unsubscription and event publication. Application has two options when using the event publication API, it can directly distribute the events to the manager's subscribers or ask the event manager to do the distribution using worker thread instead.

Audio may be missing in both directions after call is resumed from hold state, if SRTP is being used and callee answers with multiple codecs, causing us to send re-INVITE/UPDATE to lock to one codec.

The RTCP RTT calculation algorithm attempts to normalize "unusual" RTT variation, by setting the RTT to three times the current average RTT if the value is larger than this.

This ticket makes this behavior configurable by introducing PJMEDIA_RTCP_NORMALIZE FACTOR, which defaults to 3 and can be disabled by setting it to 0.

This ticket is the continuation of #1234 (missing video keyframe handling) with focus on RTCP-FB PLI for requesting video keyframe.

In PJSUA, this feature is by default enabled and configurable via call setting, i.e: pjsua_call_setting.req_keyframe_method with bitflag PJSUA_VID_REQ_KEYFRAME_RTCP_PLI. When enabled, it will be signaled in SDP offer/answer, which will look like this:

a=rtcp-fb:* nack pli

Application can also monitor incoming RTCP-FB PLI packet via PJSUA callback on_call_media_event() for event PJMEDIA_EVENT_RX_RTCP_FB.

This ticket also introduces some macro settings:

• PJMEDIA_RTCP_FB_INTERVAL: Minimum interval between two consecutive outgoing RTCP-FB packets, such as Picture Loss Indicationc.
• PJMEDIA_VID_STREAM_MIN_KEYFRAME_INTERVAL_MSEC: Specify the minimum interval to send video keyframe.