Workaround for frame bursts and drifts from audio devices

[bennylp]

Problem 1: Audio burst:

Most audio devices will call record and playback callbacks in burst, rather than in perfect interleave (for example, burst of record callbacks, followed by burst of playback callbacks, and so on). Because of this, audio buffering needs to be implemented in few pjmedia components (conference bridge, splitcomb, echo canceller, etc.), and the maximum buffering is controlled by PJMEDIA_SOUND_BUFFER_COUNT macro.

This is not ideal, because:

1. it adds audio latency of up to PJMEDIA_SOUND_BUFFER_COUNT frames, while theoretically the delay should be limited to the maximum burst level of the device.
2. it will add even more end to end latency when the sound device is connected directly to the stream (rather than via conference bridge), because of how the jitter buffer is implemented.
3. pjmedia components need to handle the burst, while this problem should have been localized and fixed in sound device.

Problem 2: Audio drifts:

Somehow related to this problem, it's common for sound devices on PC to have clock drifts (see some of our test results in Audio Device Test page). Because of this drifts, the buffering mechanism above will ultimately underflow or overflow, and this will cause a clicks noise to be generated.

The Solution:

This ticket solves the problem in several ways:

• introduce delaybuf.[hc] as a generic buffering mechanism. The delaybuf automatically learns the optimal buffering to be applied to audio flow, and shrinks or expands the audio in the buffer as necessary (for example, upon overflow or underflow situation) without degrading the audio quality by much or producing the clicks noise. This is done by using WSOLA algorithm, implemented in ticket #497
• use delaybuf in soundport, so that now recording and playback callbacks are called in perfect interleave.
• also install delaybuf in splitcomb and conference bridge since they need the same buffering mechanism.

[bennylp]

In r1664:

• Added delay buffer implementation in pjmedia. It's disabled by default for now.

[bennylp]

More work in r1715:

Changed the way sound buffering works when PJMEDIA_SOUND_USE_DELAYBUF is enabled (note: default is still disabled for now). When it is enabled, the PJMEDIA_SOUND_BUFFER_COUNT macro now will be used as the maximum delay to be accommodated by the delay buffer, and PJMEDIA components that previously use PJMEDIA_SOUND_BUFFER_COUNT value (such as conference bridge, splitcomb, and echo canceller) will set its buffer count to 1.

In the delay buffer itself, there are two problems:

• if we set LEARN_COUNT and WAITING_COUNT too low (for example, 8 and 8 in previous version), the delay buffer could not find the right value for the level. For example, in my system it set the level to 5, while the stable level value should be 6. Because it sets the level lower than the stable level, overflows occur.
• Setting LEARN_COUNT and WAITING_COUNT to high numbers (for example 4 and 48) solves above problem, but it will make learning time too long, and this is unacceptable since the delay buffer drops audio frames during learning time.

To solve the problems above, the delay buffer now adds 2 frames to the detected level, compared to just 1 frame in previous version. This setting is set as SAFE_MARGIN macro in delaybuf.c. With adding 2 frames it seems that now it can find the correct level without having to set long learning time period. So the default setting is now:

#define WAITING_COUNT  4
#define LEARN_COUNT    16
#define SAFE_MARGIN    2

[nanang]

Attachment ticket438.patch:

• added WSOLA to delaybuf, this is a great feature of time domain samples modification to compensate clock rate drift between PUT & GET.
• replaced snd_buf mechanism to delaybuf in conference, to manage passive ports buffer, so now conference should be driftproof.
• updated modules that use delaybuf, because of a small API changing of the delaybuf.
• added small part of drift case test on sound device port.

[nanang]

Attachment ticket438.2.patch:

• as attachment ticket438.patch.
• added new delaybuf API pjmedia_delay_buf_reset.
• modified conference to reset delaybuf when passive ports have no listener.
• modified delaybuf, in learning mode it will perform normal operation (instead of ignore frame on put & returning zero frame on get), after learning finished, the max delay will be set and buffer content will be adjusted appropriately.

[bennylp]

Patch ticket438.2.patch committed in r1833 with some modifications:

• added mutex
• changed some logging
• added options argument in delaybuf_create()

[bennylp]

In r1834:

• Added checking for buffer empty before calling shrink_buffer()

Thanks.

[bennylp]

Patch to add delay buffer in splitcomb

[nanang]

Added idle operation detection, delay buffer will automatically shift its state to STATE_PAUSE if idle detected

[bennylp]

Patch combination of splitcomb patch and delay buffer pause patch. With this patch, the splitcomb still have pause functionality, which we will remove

[bennylp]

In r1840:

• Incorporate delaybuf into splitcomb, with automatic suspension to media flow if the one or both side of the splitcomb doesn't supply audio frames.
• Minor change in the delaybuf logging.

[bennylp]

In r1844:

• improve documentation/comments in splitcomb and wsola.
• fix possible buffer corruption because of simultaneous thread access in splitcomb
• add channel_count parameter in wsola.
• changed expand() algorithm slightly, moving the template frame on subsequent expansion.
• changed wsola test application to repeat expansion/compression

[bennylp]

A function to be added in pjsua_app.c to instantiate multiple sound devices, for splitcomb testing

[bennylp]

In r1847:

• bug fixed, wrong param passed to shrink_buffer() in set_max_cnt(), should be (buf_cnt - new_max_cnt), instead of (old_max_cnt - new_max_cnt)
• added checking current samples number before calling shrink_buffer