wiki:Audio_Dev_API

Version 15 (modified by bennylp, 16 years ago) (diff)

--

PJMEDIA Audio Device API

Table of Contents

  1. Background
  2. Features
  3. Migration Path
    1. Common Tasks
    2. Ported Devices
    3. Compatibility settings
    4. Migration strategy
  4. Changes
    1. Default device convention
    2. Configuring AEC with the sound port
    3. PJSUA-LIB applications
    4. PJMEDIA applications

The PJMEDIA Audio Device API is a new sound device abstraction API/library in PJMEDIA, introduced in PJSIP version 1.1, deprecating the ​existing sound device API.


Background

The PJMEDIA Audio Device API was introduced as part of the implementation of APS-Direct implementation in PJSIP version 1.1. During the design and implementation of APS-Direct project, it was clear that the ​existing sound device API lacks many features that are needed to support the project. We had the choice to either patch the existing API with new features that are specific to Nokia APS, and potentially break existing applications anyway, or design a new sound device API to support all these new features as well as future enhancements, while providing some support for the old sound API and a clear migration path to the new API.


Features

The new audio device API contains the following major features.

Forward compatibility:
The new API has been designed to be extensible, it will support new API's as well as new features that may be introduced in the future without breaking compatibility with applications that use this API as well as compatibility with existing device implementations.
Backward compatibility:
While this is a completely new audio API and implementation, compatibility layers have been implemented for existing applications or sound device implementations that make use of the old API. More will be explained later.
Device capabilities:
At the heart of the API is device capabilities management, where all possible audio capabilities of audio devices should be able to be handled in a generic manner. With this framework, new capabilities that may be discovered in the future can be handled in manner without breaking existing applications.
Built-in features:
The device capabilities framework enables applications to use audio features built-in in the device, such as echo cancellation, built-in codecs, audio routing, and volume control.
Codec support:
Some audio devices such as Nokia/Symbian? Audio Proxy Server (APS) and Nokia VoIP Audio Services (VAS) support built-in hardware audio codecs (e.g. G.729, iLBC, and AMR), and this feature is supported by the new audio device API.
Multiple backends:
The new API supports multiple audio backends (may be called factories or drivers in the code) to be active simultaneously, and audio backends may be added or removed during run-time.


Migration Path

Common Tasks

  1. The Audio Device API is implemented as a new (static) library called pjmedia-audiodev, under pjmedia directory. Please add this library into link specifications of your application.

Ported Devices

The following audio device backends have been ported to the new API:

  • PortAudio
  • Windows Multimedia/WMME
  • Nokia APS for S60
  • Symbian MMF/MDA

Compatibility settings

Two compile-time macros have been introduced to enable or disable the compatibility layers. The default values for the settings enable existing application to work unchanged,

PJMEDIA_HAS_LEGACY_SOUND_API

Location: pjmedia/config.h
Default: 1
Purpose: to enable application to access the sound device using the old (sound.h) API.
Description: This macro controls whether the legacy sound device API is to be supported, for applications that still use the old sound device API (sound.h). If this macro is set to non-zero, the sound_legacy.c will be included in the compilation. The sound_legacy.c is an implementation of old sound device (sound.h) using the new Audio Device API.

PJMEDIA_AUDIO_DEV_HAS_LEGACY_DEVICE

Location: pjmedia-audiodev/config.h
Default: 0
Purpose: to enable sound device implemented using the old API to be accessed using the new API.
Description: This setting controls whether the Audio Device API should support device implementation that is based on the old sound device API (sound.h).

Migration strategy

Use the following checklist to select which setting is appropriate for your build.

  1. I use sound devices provided by PJMEDIA (I don't have my own sound device implementation), or I have ported my sound device implementation to the new API
    1. I haven't changed my application to use the new API
      • the default settings should be okay
    2. I have changed my application to use the new API
      • the default settings should be okay. You may disable PJMEDIA_HAS_LEGACY_SOUND_API to make sure that all codes are using the new API (as well as to reduce footprint slightly).
  2. I have my own sound device implementation, based on old API
    1. I haven't changed my application to use the new API
      • disable both PJMEDIA_HAS_LEGACY_SOUND_API and PJMEDIA_AUDIO_DEV_HAS_LEGACY_DEVICE. Note that with this setting, you cannot use the new API to access your device implementation. PJSUA-LIB will not work either as PJSUA-LIB uses the new API.
    2. I have changed my application to use the new API
      • disable PJMEDIA_HAS_LEGACY_SOUND_API and enable PJMEDIA_AUDIO_DEV_HAS_LEGACY_DEVICE.


Changes

As stated above, existing applications should build fine with the new changes. However, it will still be beneficial to port the application to use the new API since the old sound API has been deprecated, and also to make use of the new features in the new Audio Device API.

Default device convention

Old API
Value -1 is used throughout to denote default device
New API
Default devices are encoded with the following constants:
  • PJMEDIA_AUD_DEFAULT_CAPTURE_DEV (-1) is not used to denote default capture device, and
  • PJMEDIA_AUD_DEFAULT_PLAYBACK_DEV (-2) for default playback device.

Applications using the old convention is expected to work with the new API.

Configuring AEC with the sound port

Old API
The echo cancellation is usually configured after the sound port is created, by using pjmedia_snd_port_set_ec().
New API
The semantic of pjmedia_snd_port_set_ec() API has been changed to to change the settings of the AEC rather than to enable AEC. With the new API, the best way to configure AEC is to set the EC parameters in the pjmedia_aud_param when creating the sound device or sound device port, e.g.: 
pjmedia_aud_param param;
..
param.flags |= (PJMEDIA_AUD_DEV_CAP_EC | PJMEDIA_AUD_DEV_CAP_EC_TAIL);
param.ec_enabled = PJ_TRUE;
param.ec_tail_ms = ..;
..
pjmedia_snd_port_create2(pool, &param, &snd_port);
If pjmedia_snd_port_set_ec() is called after the sound port is opened, the behavior is as follows:
  • if the device supports (hardware) AEC, as indicated by device capabilities, then the sound port will forward the change request to the device. The device may or may not support the request to change AEC parameters after the device is opened.
  • if software AEC is being used (for devices that don't support AEC), this will change the setting of the software AEC. Effectively this resembles the old behavior of the API.

PJSUA-LIB applications

PJSUA-LIB applications that only make use of the API in <pjsua.h> should require minimum changes.

  1. Add pjmedia-audiodev library to the link specifications of your application.
  2. Latency setting. Old code may set latency directly accessing PJMEDIA API: 
    pjmedia_snd_set_latency(rec_latency, play_latency);
    Now latency setting is part of PJSUA-LIB API: 
    pjsua_media_config media_cfg;

..
media_cfg.snd_rec_latency = ..;
media_cfg.snd_play_latency = ..;
    Call to pjmedia_snd_set_latency() will only set the latency if the sound device is opened using the old API.
  3. New API:
    • pjsua_snd_is_active(): to query if sound device is currently active
    • pjsua_snd_set_setting()/pjsua_snd_get_setting(): to set or retrieve the value of sound device capability (example of capabilities are audio routing and volume control)

PJMEDIA applications

Below are list of changes for applications that directly uses the old sound API as declared in <pjmedia/sound.h>.

  1. Change the header file to include. Old code: 
    #include <pjmedia/sound.h>
    New code: 
    #include <pjmedia_audiodev.h>
  2. Add pjmedia-audiodev library to the link specifications of your application.
  3. Sound device subsystem startup and shutdown. Old code: 
    pjmedia_snd_init(pf);
 ...
pjmedia_snd_deinit();
    New code: 
    pjmedia_aud_subsys_init(pf);
 ...
pjmedia_aud_subsys_shutdown();
  4. Device enumeration. Old code: 
    unsigned i, dev_count;

dev_count = pjmedia_snd_get_dev_count();
for (i=0; i<dev_count; ++i) {
    const pjmedia_snd_dev_info *dev_info = pjmedia_snd_get_dev_info(i);
}
    New code: 
    unsigned i, dev_count;

dev_count = pjmedia_aud_dev_count();
for (i=0; i<dev_count; ++i) {
    pjmedia_aud_dev_info dev_info;
    pj_status_t status;

    status = pjmedia_aud_dev_get_info(i, &info);
}
  5. Opening sound device. Old code: 
    pjmedia_snd_stream *stream;
pj_status_t status;

status = pjmedia_snd_open(-1,       // default capture device ID
                          -1,       // default playback device ID
                          8000,     // clock rate
                          1,        // channel count
                          160,      // samples per frame
                          16,       // bits per sample
                          &rec_cb,  // capture callback
                          &play_cb, // playback callback
                          user_data,
                          &stream);
if (status != PJ_SUCCESS)
   ..
    New code: 
    pjmedia_aud_dev_param param;
pjmedia_aud_stream *stream;
pj_status_t status;

status = pjmedia_aud_dev_default_param(PJMEDIA_AUD_DEV_DEFAULT, &param);
if (status != PJ_SUCCESS)
   ..

param.dir = PJMEDIA_DIR_CAPTURE_PLAYBACK;
param.rec_id = PJMEDIA_AUD_DEV_DEFAULT;
param.play_id = PJMEDIA_AUD_DEV_DEFAULT;
param.clock_rate = 8000;
param.channel_count = 1;
param.samples_per_frame = 160;
param.bits_per_sample = 16;
.. // put additional settings here

status = pjmedia_aud_stream_create(&param,
                                   &rec_cb,
                                   &play_cb,
                                   user_data,
                                   &stream);
if (status != PJ_SUCCESS)
   ..
  6. Latency setting. Old code: 
    pjmedia_snd_set_latency(rec_latency, play_latency);
    New code: 
    // Latency setting is now part of stream creation parameter
pjmedia_aud_dev_param param;

...
param.flags |= (PJMEDIA_AUD_DEV_CAP_INPUT_LATENCY | 
                PJMEDIA_AUD_DEV_CAP_OUTPUT_LATENCY);
param.input_latency_ms = ..;
param.output_latency_ms = ..;

status = pjmedia_aud_stream_create(&param, .. );
  7. Device routing. Old code (supported on APS only): 
    pjmedia_snd_aps_activate_loudspeaker(stream, PJ_TRUE);
    New code: 
    int value = PJMEDIA_AUD_DEV_ROUTE_LOUDSPEAKER;

status = pjmedia_aud_stream_set_cap(stream, 
                                    PJMEDIA_AUD_DEV_CAP_OUTPUT_ROUTE, 
                                    &value);