Ticket #438 (closed enhancement: fixed)

Opened 9 years ago

Last modified 9 years ago

Workaround for frame bursts and drifts from audio devices

Reported by: bennylp Owned by: nanang
Priority: critical Milestone: release-0.9.0
Component: pjmedia Version: trunk
Keywords: Cc: nanang
Backport to 1.x milestone: Backported:

Description (last modified by bennylp) (diff)

Problem 1: Audio burst:

Most audio devices will call record and playback callbacks in burst, rather than in perfect interleave (for example, burst of record callbacks, followed by burst of playback callbacks, and so on). Because of this, audio buffering needs to be implemented in few pjmedia components (conference bridge, splitcomb, echo canceller, etc.), and the maximum buffering is controlled by PJMEDIA_SOUND_BUFFER_COUNT macro.

This is not ideal, because:

  1. it adds audio latency of up to PJMEDIA_SOUND_BUFFER_COUNT frames, while theoretically the delay should be limited to the maximum burst level of the device.
  2. it will add even more end to end latency when the sound device is connected directly to the stream (rather than via conference bridge), because of how the jitter buffer is implemented.
  3. pjmedia components need to handle the burst, while this problem should have been localized and fixed in sound device.

Problem 2: Audio drifts:

Somehow related to this problem, it's common for sound devices on PC to have clock drifts (see some of our test results in Audio Device Test page). Because of this drifts, the buffering mechanism above will ultimately underflow or overflow, and this will cause a clicks noise to be generated.

The Solution:

This ticket solves the problem in several ways:

  • introduce delaybuf.[hc] as a generic buffering mechanism. The delaybuf automatically learns the optimal buffering to be applied to audio flow, and shrinks or expands the audio in the buffer as necessary (for example, upon overflow or underflow situation) without degrading the audio quality by much or producing the clicks noise. This is done by using WSOLA algorithm, implemented in ticket #497
  • use delaybuf in soundport, so that now recording and playback callbacks are called in perfect interleave.
  • also install delaybuf in splitcomb and conference bridge since they need the same buffering mechanism.

Attachments

ticket438.patch (21.7 KB) - added by nanang 9 years ago.
ticket438.2.patch (24.2 KB) - added by nanang 9 years ago.
splitcomb.patch (17.6 KB) - added by bennylp 9 years ago.
Patch to add delay buffer in splitcomb
splitcomb2.patch (17.4 KB) - added by bennylp 9 years ago.
delaybuf.patch (6.1 KB) - added by nanang 9 years ago.
Added idle operation detection, delay buffer will automatically shift its state to STATE_PAUSE if idle detected
scomb-dbuf3.patch (23.3 KB) - added by bennylp 9 years ago.
Patch combination of splitcomb patch and delay buffer pause patch. With this patch, the splitcomb still have pause functionality, which we will remove
multiple_snd_devs.c (4.1 KB) - added by bennylp 9 years ago.
A function to be added in pjsua_app.c to instantiate multiple sound devices, for splitcomb testing

Change History

comment:1 Changed 9 years ago by bennylp

  • Description modified (diff)

In r1664:

  • Added delay buffer implementation in pjmedia. It's disabled by default for now.

comment:2 Changed 9 years ago by bennylp

  • Cc nanang added

More work in r1715:

Changed the way sound buffering works when PJMEDIA_SOUND_USE_DELAYBUF is enabled (note: default is still disabled for now). When it is enabled, the PJMEDIA_SOUND_BUFFER_COUNT macro now will be used as the maximum delay to be accommodated by the delay buffer, and PJMEDIA components that previously use PJMEDIA_SOUND_BUFFER_COUNT value (such as conference bridge, splitcomb, and echo canceller) will set its buffer count to 1.

In the delay buffer itself, there are two problems:

  • if we set LEARN_COUNT and WAITING_COUNT too low (for example, 8 and 8 in previous version), the delay buffer could not find the right value for the level. For example, in my system it set the level to 5, while the stable level value should be 6. Because it sets the level lower than the stable level, overflows occur.
  • Setting LEARN_COUNT and WAITING_COUNT to high numbers (for example 4 and 48) solves above problem, but it will make learning time too long, and this is unacceptable since the delay buffer drops audio frames during learning time.

To solve the problems above, the delay buffer now adds 2 frames to the detected level, compared to just 1 frame in previous version. This setting is set as SAFE_MARGIN macro in delaybuf.c. With adding 2 frames it seems that now it can find the correct level without having to set long learning time period. So the default setting is now:

#define WAITING_COUNT  4
#define LEARN_COUNT    16
#define SAFE_MARGIN    2

comment:3 Changed 9 years ago by bennylp

  • Status changed from new to assigned
  • Description modified (diff)

comment:4 Changed 9 years ago by bennylp

  • Priority changed from normal to critical

comment:5 Changed 9 years ago by bennylp

  • Owner changed from bennylp to nanang
  • Status changed from assigned to new

Changed 9 years ago by nanang

comment:6 Changed 9 years ago by nanang

Attachment ticket438.patch:

  • added WSOLA to delaybuf, this is a great feature of time domain samples modification to compensate clock rate drift between PUT & GET.
  • replaced snd_buf mechanism to delaybuf in conference, to manage passive ports buffer, so now conference should be driftproof.
  • updated modules that use delaybuf, because of a small API changing of the delaybuf.
  • added small part of drift case test on sound device port.

Changed 9 years ago by nanang

comment:7 Changed 9 years ago by nanang

Attachment ticket438.2.patch:

  • as attachment ticket438.patch.
  • added new delaybuf API pjmedia_delay_buf_reset.
  • modified conference to reset delaybuf when passive ports have no listener.
  • modified delaybuf, in learning mode it will perform normal operation (instead of ignore frame on put & returning zero frame on get), after learning finished, the max delay will be set and buffer content will be adjusted appropriately.

comment:8 Changed 9 years ago by bennylp

  • Owner changed from nanang to bennylp
  • Status changed from new to assigned
  • Summary changed from Workaround for frame bursts from audio devices to Workaround for frame bursts and drifts from audio devices

Patch ticket438.2.patch committed in r1833 with some modifications:

  • added mutex
  • changed some logging
  • added options argument in delaybuf_create()

comment:9 Changed 9 years ago by bennylp

  • Owner changed from bennylp to nanang
  • Status changed from assigned to new

comment:10 Changed 9 years ago by bennylp

  • Description modified (diff)

comment:11 Changed 9 years ago by bennylp

In r1834:

  • Added checking for buffer empty before calling shrink_buffer()

Thanks.

Changed 9 years ago by bennylp

Patch to add delay buffer in splitcomb

Changed 9 years ago by bennylp

Changed 9 years ago by nanang

Added idle operation detection, delay buffer will automatically shift its state to STATE_PAUSE if idle detected

Changed 9 years ago by bennylp

Patch combination of splitcomb patch and delay buffer pause patch. With this patch, the splitcomb still have pause functionality, which we will remove

comment:12 Changed 9 years ago by bennylp

  • Status changed from new to closed
  • Resolution set to fixed

In r1840:

  • Incorporate delaybuf into splitcomb, with automatic suspension to media flow if the one or both side of the splitcomb doesn't supply audio frames.
  • Minor change in the delaybuf logging.

comment:13 Changed 9 years ago by bennylp

  • Description modified (diff)

comment:14 Changed 9 years ago by bennylp

  • Description modified (diff)

comment:15 Changed 9 years ago by bennylp

In r1844:

  • improve documentation/comments in splitcomb and wsola.
  • fix possible buffer corruption because of simultaneous thread access in splitcomb
  • add channel_count parameter in wsola.
  • changed expand() algorithm slightly, moving the template frame on subsequent expansion.
  • changed wsola test application to repeat expansion/compression

Changed 9 years ago by bennylp

A function to be added in pjsua_app.c to instantiate multiple sound devices, for splitcomb testing

comment:16 Changed 9 years ago by bennylp

In r1847:

  • bug fixed, wrong param passed to shrink_buffer() in set_max_cnt(), should be (buf_cnt - new_max_cnt), instead of (old_max_cnt - new_max_cnt)
  • added checking current samples number before calling shrink_buffer
Note: See TracTickets for help on using tickets.